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It is most eaeouraging to note ^^H^ng growth and success 
of ft© Invitational Conference on TCT^^groblems, particutarfy 
whtn wa recaU ftat tte first ^nferecSS^eW in 1936, atfracted 
a tptaj of seven testing leaders 1 Thfel^rr'the Conference brought , 
tbgeftter" ^ost ^ persons from aiM^ tiie United States and 
from saveral foreign c^nprte|. i^^^ ' ' 

Ha expanded program, judgkigy^p tiie responses of tiiose 
attending the Conference; appears ti^Ifave been very well re-? \ 
ceived, du§ in vpry large* part to the ceinpetence of the speakers 

^and Ae pxceUence of tiieir papers* E'fS is particularly apprada- . 
tiye of: their participation' in the j^nf erence and tteir contribu- * 
tion to iti siifiee^s, ^ i 

We Me indeed deeply indebted to waiter Dtj^sf for his OTpert 
di^ij^atiship o&A Gonference and for^e imagiftfltion and 
energy wifli .wnich he planned and conducted the pfo^am. To 
him goes dievfuU credit fpr thq', enthusiastic response to the Con- 
ference and with it tiie sincere thanks of ETS for a most success- 
ful meeti^. r . ^ « ; . 

■ . ! ^ ^ ■ Henry CHAtm^Y ■ / 

■ J , V President 
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EREFACEt , 

Waof It \ ^ plan the 19^, Invftatjonal Gonfarinca on 

Testing B^l)Ims; piii^ stood out b^i&g^pf parapiount im- 
por^D^ n^elji"^ Mat ttib;^ai np* longer a sAiaU gadiarmg of. 

tation/ list m(^#d nearly 1^000 nai^es md the fields ^ ape- ' ■ 
c^Uiation I rapr^^tad, whda ha\Ag ^^urament overtones, 

neVtri^etes v^^vej^iflW. For ^is i^son, ^e plan waa v 
Idt upon of dividiQg/Up die seoopd hour into five saotional meat^ ^ 
usgs, plamaed 'i^paciaUy to tatar to mora spadalizad interests. ' 
Cotmting genial sassioni ^nd the luncheon meeting a total 
^ $3^pa^ wm Tp^mt&i: la all djapst SOO p^oos attrodad 
eonf^anoa and a substwtial n\|i^Rty of those quastionadr 
ap^bved of Ais, program divtt-siflcmtion. - 
i/rha substantially larger numbar of papers arid hOTC© tfie . 
greater cost of producing dia Procaedirigs poiad ahotiiar problem, 
which has been mat ^is yegrjby mateig a modestj£arge f or the 
yoluna. ?liis has Qxe added advantaga of makingjme Procaedings 
mora ganarally available and> to view of &a exoellenca of the 
fNpipars prteantad and fteir periaianent vgjue, Ais is especially 
dasirable. TTia lu^chjeon addrass 'by D^. I^man L. Kalley was 
ddVotad to ^a desOTption of some new test developments in fta 
field* of interest tasting* Dr* Kelley spoke extemporaneously and 
presented in an ihfomalfmannar some very tenjativa data de- 
scribing his Activity PreferenA Inventory*; Bacause of the in-", 
foraaal ^atoa of his reportj as well as tiie tentative natur^of ttia 
findings described^ Dr* Kelley has requested that his address be 
omitted from ^e Proceedings* To ^is the Chairman and Educa- 
tional Tasting Sarviea have reluc^ntly agreed but only because 
wa have been assured that furtiier pubhcation describing his 
work in tiiis field is planned for ttie near future* 

Ha Oh^mto would like to take tiiis opportunity to egress 
once again his deep appreciation to me speakers^ the chairaienj 
and to Educational' Testing Service for tiie 100 per cent coopera'- 
tion received on evety jpoint in tiie development and implementa- 
tion of this Conference, , ^ f 

; . Walt^ N* Duiost^ Cftflinnani V 

^ , . 1953 Conference^ ) \ 

* * » 
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REMARKS^ OF TH^ CHAIRMAN 



One 'tou: a few years ago* on the spur 
of momentp I daflned an evaluator 
as an expert in measurement wi& A 
soeial consGienca* ^Our^ire© speakers 
this morning haval demonstrate tiiat 
thete may really ^e a kernel of fruth in 
thiS'-deflnition. ■ 

Evaluation is always concerned with 
three things? , 

It is conaemed &st with philosophy, 
^^at are the objectives? What are the 
goals? Toward what end^ are education 
and evaluation directed? 

It is* concerned, second^ with the 
science of measurement— with such 
problems * as rehability, Yahdityj rele- 
vance, cornprehensiveness, discrimina- 
tionj and sampling, 

It is concerned^ thirds with the psy- 
chology of human relations, learning, 
ajid personality, What modes of action 
mp most likely to produce changes in 
behavior? What interpersonal and in- 
tergroup relationships musf' be con- 
sidered? What is the evaluation .divi- 
dend for me prpfessor and his teaching, 
for ttie student and Bis Ipaming, and 
for the instiAition and the effectiveness 
of its pro-am? # ^ 

'Pediaps, UJ^orturiately, over the past 
^o dec^deSi much that is pertinent for 
evaluation has tended to b^e produced 
separately in education, psychology, 
and measurennent, and with what Dr, 



^esii^ judges to be a widening, gap i 
in understanding* 

Many wUege professors and educa- 
tional phil^ophers have been oitieizing 
the failures of higher #ducad9n in 
sweeping generalizations unsupported 
by systematic objective evidence, and 
finding re%ige in what se^s to me the 
curiously anti-intellectual notion that 

"Anything which pan be measured, or 
even dealt with empiricallyj cannot be 

^ very important, * 

Many of the'fenajor advances in meas- 
urement theory and technique in re- 
cent years have come out of statistical 
laboratori^j national testing agencies 
and researdi bureaus* And many of tfie 
' ^good people ii^^these environments ap- 
pear to be divorced from the on-going 
responsibihties of teaching and leam- 

w ing, and from direct and daily contact 
with the procisses of higher education, 
Many of the recent major research 
eflForts in psychology concerned with 
human relation^ group behavior and 

. decision making, personality, and atti- 
tude change have been conducted in 
the context of industry^ govemmentj 
the armed forces, and voluntary com- 
munity p'oups — again appearing to be 
divorced from the framework of higher 
^ducation and its evaluation, Indeed, I 
suspect that the experience gained inP 
cooperativf educational projects^ — ^such 
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as tha'Ei^VYear Study, the Coopera- 
tive Collage Study of General Educa" 
tion* lie Commission on Teacher Edu- 
cation, or the recent Cooparative Study 
_ of Evaluation in General Education— 
^ \ ii largMy unknown by many of the 
piyc hologiits^ who are now weatling 



with similar problems ^ a diffCTent 
level of theory and research^ and in a 
different context. 

If we are to piake a major improve- 
ment in the evaluatioh of educational 
outcomes at tiia college l^el, we need 
to bring these related groups and these 
related problems into better focus. 



The concept of systemi research 
could be productive at ^is point if it 
forced our attention on tW necessity of 
integration* * _ 

W}ien ^valuation is seen as contrib- 
uting to the improvement of learning, 
of insfruetion, and of the efforts of the 



institutfoh as a^wKole to aftlin~itI~ob— 
jectivesi and when it is seen as a neces- 
sary integration of philosophyi science 
of measurement^ and psychology of hu- 
man relations, it may be well on tiie 
road toward a significant development 
in behavioral science and a major ad- 
vance in education* 
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HpW AN EXAMII^T10N*SERVICE HELPS 
COLLEGE TEACHMS TO GIVE BETTER TESTS 



II Descriptive Bacxgrqund 

Mait 1 %B0n by giving. a brief de- 
SOTption of the ExaminatiDns .Service of 
the Stale Universi^ of lowfe. It was 
established April, 1943 under Ae 
lead^hip of ftofesor E, F. Lindqui|t. 
The first director of fiie seryice. Prof Is- 
sor Paul Blonmars^ was largely re- 
sponsible for 6rganizing and equipping 
it, aQdflor setting up its procedures. 

CurreaUy the Exarninatiorii Service 
occupies several small rooms^^in tiie 
principal office building of tiie univer- 
sity/ T^fe staff 'conjists oL«^en clerical/ 
ttaployej^^ a graduate assistant^ and 
the director. Itfultilith duElicatori, iflSt 
icoiing marlines ^d typewriterSj^^Mon- 
roe caloulators, an'fi ie usual d^ksj 
filing cabi^tSi and storage cupboards 
constitute die principal items of equip- 
^ment 'Hie service operate on an an- 
nual budget for salaries ahd supplies of 
approximately $30,000. 

One of the principal functions of the 
Examinatioi^ Ser^ce is to relieve uni- 
versity instructors of die mechamoal 
and clerical burdens involved in duph- 
cating, scormgi and ^analyzing course 
laminations. Each year a few ekamina- 
tidns are constructed as;special projectsv 
but this is not one* of the mam functions 
of *the service, Its principal responsi- 
bility, is to provide, upon request, as- 
sistiLnce and advice on technical prqb- 
lemMi tovolvjed in tibe construction, 



aSminis^^flros ,and interpretation of 
exarninations of all types* Another area , 
of responsibility j's the adminisbation of 
special testing* programs, Such as^en- 
trance examinations and various 
achievement testing programs. Special 
^ individual testing for the purposes bf 
^ guidance^nd counseling is the functidn 
^ of^ separate o£Bce. r 

IL PfioCEDunAL Stuategy ' 

The long-range goal of the Examina-" ' 
tions Service is to contibute to tfie 
effectiveness of the universityj educa- 
tional programs by improving the qual^^ 
ity of the testing done. The sttategy fo^ ^ 
achieving this objective involves three 
main considerations. The first is to 
make it as convenient as possible for 
college^ instructors *to give good tests. 
No charge is made for any of the serv- 
ices offered. The dfflces are centtally 
located on campiis. Service is rendered 
as promptly as possible hnd pains are 
taken-to prevent errors. * 

The second consideration is, to avoid 
afiy suggestion oi interference with the 
instructor's independence with respect 
to his own examination procedure, 'pie* 
service is operated on a voluntary, per- ' 
missive ba^s. If an instructor choosia 
to give a speed test when a power test 
seems more appfopriate, or -to ciH for 
raw scores reported as 100 less thi 
number of eirors, radier tfian'Ss Ae 
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.nuriiber o&con'ect answers, or to weight 
each tnu^le-choice item 4 times as 
mucib as eaeh bme^fdse item in his test, 
we may inquire the reason for his pro- 
dedure, but it is his pTefer|nce, not 
ou^, which determines how the job 
will be done. * . ' * 

It may seeA somewhat inconsistent 
that a service established to ^ improve-^, 
examination practices should cooperate, 
in carrying out *unihiproved practices ' . 
.Yet it appears that helpful service offers 
^)theftbest opportunity^ fpr gradual im- 
provement, College professors are hu- 
niB^ and cherish their feelings at ade- 
quacy and., independence. 'However 
much their methods ^eed mending, 
they do |iot like to be told so directly. " 
'Further they fe^ some distrust for 
those they call ^'educationists/' Many of 
us who work in the-field of educational 
measGremerits have- backgrounds of 
public school teaching or hold degrees 
from colleges of education. We could 
not ' succeed even if we ti^iedj ii^ irn- 
posing^ upon themj our ideas of what 
is right and proper in educational meas- 
urement, 

Tlie tl^trd considerktion is Id main- 
tain free and friendly communication 
with staff members. They receive no- 
tices regarding^he e.^tent and a vii li- 
ability of the service.. They are sent 
bulletins dealing with such matters as 
item writing, test adrninistrationi and 
the interpretation of item analysis data. 
" Opportunities to consider examination 
problems with depnrtniental faculties 
and with individual stuff members are 
welcomed. 

III. Basic PnoBLEMs ) 

Having given this brief description 
of our organization and service, may 1 
turn next to consideration of some of 
the basic problems we have encoun- 
tered in our efforts to help college 
teachers give better tests. It is emphati- 
cally asserted by wounded students and 
harassed professors and generally agreed 



by impartial observers, tliat college 
testiilg practices need improvement, In 
a recent survey of student opinions of 
teaching in our College of Liberal Arts, 
the item on whick -insfamctors were 
ranked, lowest w^s "Quality of Exami- 
natiofft." This low rating may be ex- „^ 
plained in part as defensive ratiorializa= 
tion by students vvho received lower 
"grades than they expected or desired. 
But, f am convinced, it also indicates 
that there is very great room for inir 
provement in man/ of our* classroom 
tests. WTiat ^has^ stodd in the way^ of^ 
,ffiore rapid impro\^ement? It seems -to 
me that certam common attitudes and 
i^isconceptions on the part of college 
teachers constitute thB chief barriers. 
- To begin with there are some pro- 
fessors who do not take the responsi= 
bOity of evaluating student achieve- 
ment seriously. They regard testing and " 
griding as non-ess*entiai administrative 
red\ tape, largely divorced from the 
essential process of education. Since, in 
their -vieWi the whole process is unim- 
portarit, it is unnecessary for them to 
expend much time or effort to achieve 
high validity or precision in the proeess. 
Such professors often sotve Uie problem 
of grading according to the principal 
of least \innoyance. "Give few low 
grades and you will liavc few com- 
plaints from the students, and hence 
there will be few occasions for outsiders 
to ciucstion your grading procedure/' 

It is interesting to note that the same 
professors who. shun the onerous tasks 
of accurately' evaluating student 
achievement in their own classes are 
hkely, quite inconsistently, to object 
very strenuously to thc'consequences of 
similar behavior by Other teachers. "My 
students can't read or write," they cry 
in anguisli. 'TIow did they ever pass 
freshman English?'' Sometimes a pro- 
fessor finds himself in this unhap]^* 
predicament. Ho is serving on the doc- 
toral examination committed of a very 
weak candidate. Ready to. object stren= 
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uously, and possibly to cast a negative 
vote, he suddenly discovers thati in the 
''5' recent pastj he has himself rewarded 
the student's mediocre talents with a 
grade of "A," 

The. second obstacle to the improve- 
ment of eoUege tests is the' refusal of 
many professors to ftcknowledge the 
importancl of objectivity rri the evalu- 
' ation of achievement. Note that we did 
not say, "the importance of objective 
tests/' Such testi are* valuable but they 
do not provide the only pathway to ob- 
Jective measurement. Evaluation is ob- 
jective to the degree that equally com- 
petent observers can agree in their 
evaluatu)n of a particular achievement. 
Our thesis here is that no measurement 
or evahiation^ however ob^ined^ is 
wOrUi anything if it Js not objective in 
some degree. To put it another way^ 
only that component of a measurement^ 
or evaluation which can be verified by 
independent observation can sen^ any 
useful purpose, 

- Strange as it may seem, there ^sts 
a considerable group among college 
professors who reject the idea that ob- 
\ jectivity ii an essential^ ingredient in 
good evaluation. Two subipecies of this 
group may be recognized. The first is 
composed of those who worship the 
mystical goddess of intangibility. The 
second, consists of those who are certain 
that any judgment which differs from 
their own must be wrong. 

Here is a professor %vho bases his 
evaluation on a student's extended di.i- 
cussion of a rather vague topic. He pre= 
fers this procedure because, as he says, 
it yields occasional flashes of insight 
into how the student s mind is working. 
If these flashes please him* he is likely 
to give the student an A, It bother^ him 
not at all that an equally competent 
professor^ failing to receive these same 
flashes, or' valuing them lesSs might give 
the same paper a C. He may know that 
his grade of A will be placed on record 
together with the A's assigned by a 



ROBLEMS 5 

great many other insfructors, and that 
when such grades are combined to give, 
an overall indication of achievement,' 
each '*A" is assumed to have essentially 
the same mea^Mig as any other "Ap" But 
the obvious inference from these facts 
that some sort of common" standard! 
(objectivity) in grading are reqnired 
seems to have escaped him. Our Exani" 
inations Service do'es very little in. the ' 
way of analysis of essay tests^ not be- 
cause such analyses would be impossi- 
ble to performs ,but rather,, because 
many users of essay tests are subjec- 
tivists who fail to see any significance 
in such analyses. * 

A third major obstacle to the im- 
provement of college tests lies in the 
emphasis upon testing for. rec^aU of 
course content and the neglect of test- 
ing the attainment of course objectives. 
Some of the poorest examinations we 
handle come from the older academic 
fields where students are simply^ asked 

^ to recall this fact or that stat^aent 
which was presented dufing the course. 

' Some of the best tests come from"* pro- 
fessors of medicine, and law. Concrete 
case situations are descrij^ed and the 
examinee is required to hiake decisions 
concerning treatment or procedures. 

We hold it to be' a fundamental 
principk of good educational achieve- 
ment testing that a test should measure 
as directly as possible as many as possi- 
ble of the ultimate objectives of in- 
struction in the course. We have little 
patience with those who assert that 
many of the important Outcomes of in- 
struction are intangible^ 'Does this mean 
that no one can obsejve them? If so 
they cimnot possibly be of any impor- 
tance, except in the internal life of the 
particular individuak And none of us 
can possibly be concerned about tlie in- 
ternal life of another individual ^except 
when it manifests itself in overt be- 
havior, I strongly suspect that many 
those who insist upon die importance^ 
of intangible outcomes of education are 
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simply using it as a iWeld for theu' re- 
luctmice, or inability, to describe spe- 
cifically what a given course of Insfruc- 
tion ought to ac^mplish. 

Many college instructors have seri= 
ously limited and distorted notions con- 
cermng what can be measured in well 
eonsmiet^ t^ts of educational achieve- 
ment* Having had limited experience 
with objective type questions they as- 
sume ^at only the recall of factual de- 
tails, or, worse stiU, die Srecognitions of 
names and verbal symbols, can be 
tested with such a device. You fre- 
quently hear it asserted tiiat *'we do not 
yet know how to measure certain Im- 
portant outcomes of education," This 
may be ^le, but I am convinced that 
most of the difficulty is due to inabihty 
to define precisely what is to be meas- 
ured. I learned recently of a college Jn= 
sfructor who was seeking help in meas- 
uring leadership potentiaL He felt ,the 
measurement specialists were being 
evasive and uncooperative when they 
persisted in asking hta precisely what 
he meant by "leadership potential" 

A fourth obstacle to the Lmprovement 
of college tests arises from the reluc- 
tance^ of many professors to recognize 
the inherently relativ-e character of 
most measurements of educational 
achievement. Many of them feel quite 
strongly that ^e standards they should 
employ are set by the subiect matter 
itself, and are best interpreted and ap- 
plied by one who possesses expert 
knowledge of that subject matter. They 
would deny that the best basis for 
Judging wheAer a given student's 
achievement is superior or inferior is a 
comparison of his achievement with 
diat of other students in the same class, 
course, or ^ade group. The latter type 
of relative evaluation they refer to, 
somewhat disdainfully, as "grading on 

tiie curve*" _ t^i ' u 

This preference for presumably ab- 
solute external standards seems to be 
based on several misconceptions. One is 



that standards of achievement are in- 
herent m die subjecf^matter. If Uiey 
are, tiiey ^e ex&emely elusive. Re- 
peated attempts to iet minimum s^ndf 
ards of achievement in various subjedt F^^ 
matter areas seem to have demon- 
strated 'quite convincLngly that such 
standards ^e arbifrary, dependmg - 
largely on the preferences and judg- 
ments of 4he individuals concerned with 
establishing them. 

A second mlscondeption is that stand- 
ards set by an expert Lnsfructor are 
likely to represent more stable stand- 
ards than those 'based on poup per- 
formance. AgaLn, experience has dem^ 
onsfrated quite clearly that equally 
competent instructors in the same sub- 
ject matter have quite different stand- 
ards both qualitative and quantitative. 
Furth^, any given insttuctors stand- 
ards are iLkely to shLft markedly as time 
passes, ' ill 

It is easy for an insbuctor to deiuae 
himself about the absoluteness of his 
standards when he relies on subjective 
processes of evaluation. So long as essay 
tests prevailed an instructor could claim 
absoluteness and still see to it that not 
too many students ^failed. His standard 
of tO% correctness for a passing ^ade 
seems tp be independent of proup per- 
fomance, but actually may not be. If 
too many students seem to be getting 
scores below 70, the' instructor can 
quietily shift his basis for calculation 
witliout letting any one eba know about 

it.y^ 

With objeQtive tests, however, such 
a readjustment is much .more obvious. 
The dechne of percentage grading (and ^ 
the 70% pass mark) following the 
growth of objective testing waf no acci- 
dent. It came about, in part at leas% 
because of the great difficulty of build- 
ing an objective test which a ^nsistertt 
and reasonably large nurnber of stu- 
dents could pass when tiie minimum 
was set at 70% correctness. 

Another misconception is that of rela- 
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Uve standar4s based on student per- 
formanee ^end to be too low. The use 
of ralfttive standards does not, however, 
imply fixed perceniag^ of A's, or F*s, 
or any odier grade. Th^^ proportions 
can be set ind^endently of the group 
^ performance. Tlw j^oint is that con> 
iideratioa should be %iven to that per- 
fomanee so the standard wiU not be 
too hi^ or too low. Others have feared 
that- relative gratiling wiU enc^tfag^ 
aeademio "slo^-downs." Student^might 
say to one anptherp "If none of us per- 
fonns at c^f^city it will be easier for 
all of us to pass.*' So far as I know tfiere 
is no empirical evidence to support this 
argument, and even the logic of it ap- 
pears shaky wh|^f^^^ cohsiders die 
strength and pr^Mince of individual 
aspirations toward academic *achieve- 
ment. 

Four obstacles to more rapid im- 
provement of classroom tests a^ the col- 
lege level have been mentioned. These 
are (1) failure to recognize tlie impor- 
tance of measurement, (2) failure to 
see the need for objectivity, (3) 
emphasis on content details arid (4). 
preference for allegedly "absolute" 
standards of achievement. These are 
essentially problems pf attitude and ori- 
entation. But there is another serious 
obstacle or problem of a d^erent type. 
It is lack of knowledge of appropriate 
techniques of measurement and lack 
of skill in their application. May we 
^ consider briefly what an examinations 
service can do to help solve this prob- 
lem. 

IV, Test Analysis 

Detailed test analysis has proved to 
be one of .the most useful avenues for 
progressive improvement of the tests 
consfructed by classroom instructors. It 
possesses the two very important char- 
acteristics of objectivity and imperson- 
ality. Following test analysis it is not 
necessary for the test specialist to make 
critical cotnTnents concerning the test. 
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Such criticisms are obvious and implicib 

in the data presented/ Instead of plac- 
, ing the test specialist in the role of 
yudge and' critic, the jise, of test analy- 
sis places him in Ihe role of consultant 
and adviser, working with tiie insfruc- 
tor tO iEolve a coriimon ^problem. 

"You have been issued sample bopies 
of some of the materials we us^ in the 
test analysis.* Mafif direct your 
tention first to th^lest analysij repo^ 
form. You will note that two major test 
characteristics^ relevance and discrimi- 
nationj are the subject of analysis. 
These ar^ the most important qualities 
which an educational achievement test 
ca^? possess. Most of the suggestions for 
in^proved test consfruction relate to 
these two basic quafities either dirtily 
or indirectly. In order to ^prove the 
relevance of their tests, insfructors are 
urged to write items based on coursi 
objectives rather than on content de- 
taiLSi to emphasize useful and important 
information rather than trivial or eso- 
teric details, and to test the students 
understanding and ability , to apply 
rather than his recognition or recall of 
details. When item wit«^ are urgrf to 
choose questions of moderate difflculty, 
^o express them clearly, to make sure 
that there is onk best response, but that 
each of the alternative answers has 
spme ^basis for plausibility, thCipurposa 
is to improve the discriminating power 
of the indiyidual items and hence the 
test as a whole. ' ■. ^ 

Evaluation of the relevance of a test 
is a difficult matter vvhich is largely a 
responsibility of the subject matter spe- 
cialist ratlier than of the test specialist. 
'However, it is possible to . roughly 
classify the items with respect to the 
type of achievement they eall for, on 
the basis of very little subject matter 
competence. This is what we attempt 
to do in our analysis of relevance. Six 



• Materials disttibuted at the ponference 
are reprinted at the end of this address. 
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categories of relevance are recognized: 
^tent details, vocabulary, facts, gen- 
eralimtions, understanding, and appU- 
cations. These constitute an ascending 
scale of values. That is, we regard items 
dealii^^th tiie understanding and ap- 
plication as being mucTi more valuable 
than Aose dealing in general with con- 
tent details or vocabulary, While it is 
often difficult to be sure where a par- 
ticular item ought to be classified, the 
usual effect of the classification process 
js to ihow quite clearly where the em= 
' phasis, of the test as^ a whole lies: One 
of the main purposes of Uiis analysis 
for relevance is to make each instructor 
directly aware ' of the desirability of 
writing^ n^ore < items which deal with 
generalizations, understandings and ap-.j 
plications. 

The so-called ideals listed Jn the sec= 
ond columil of the test analysis report , 
are practical ideals rather than "ideals" 
ideals. Their purpose is to indicate the 
desired emphasis without giving any 
, instructor the" impression that his' test 
is hopelessly inadequate. The second 
sheet in the handout is a relevance 
workaheet filled in with data frpm a 
■ specific test. A copy of this dassifica^ 
tion of items is returned to the instruct 
tor along with the test analysis report 
giving the over-all distribution of em= 



The lower half of the test analysis 
'report sheet is concerned with the dis- 
crimination of individual items and of 
the test scores as a whole. The index of 
item discrimination used in this report 
is the U^L index Suggested by Johnson. 
.It is based upon the number of correct 
respdnses to an item in upper and lower 
27% criterion groups, and is .defined as 
the difference between the number of 
successful responses in upper and lower 
groups divided by the maximum possi= 
ble dLfference. It an index which 
favors items of near 50% difflculty, but 
for most tests of educational achieve- 
ment this is an advantage rather than a 



h.^ndieap. Separate indices of item di^- 
culty are reported to the instructor, to 
M him in analyzing the causes of low 
discriminating power amoog certain 
items, but they are not used as a bas^ 
for selectmg the best items or of iridi- 
eating the overfall quality of the items. 
This is done^on the basis of die U-L in-^ 
dex alone. One should remember thar 
the frame of reference for this analysis 
as cla|srbom testing. In wide-^ale 
achievement testing programs, where 
tests are designed^ to cover a range of 
grades, more attention would certainly 
need to be given to the distribution of 
item diflRculty values. 

If all of the items in the t^st possess 
high relevance to the objectives of in- 
struction, then the oply other necessary 
quality for^the test scores as a whole is 
reliability. It is well known that tlie 
standard' deviation (or variance) of test 
scores is an important factor in the re- 
liabihty of those scores, and that the 
gencmh level of scores, as indicated by 
the mean,' has some bearing on the vari- 
ance obtainable. Data on all of these 
matters are reported to the instructor. 
A copy of the score analysis worksheet 
is included in the handout. The reha^ 
bility coefficient it calculates is based 
on Aiigors simplification of Kuder^ 
Richardson formula 8. At least one 
competent ntatistician has questioned - 
the superiority of this formula over the 
more familiar Kuder^Richardson form- 
ula 20 On the other hand, this form.ula . 
. gives results identical to those obtained 
from K.R. 20 when all items are equally 
difficult. When the items differ in diffl- 
cuhy this formula has the advantage of 
an upper limit of LOO, while that of 
K.R. 20 is less than 1.00. At the mo= 
ment I am prepared td argue that An- 
* gi.ffs formula provides the best reason^ 
Libly convenient estimate of test ^core 
reliabihty that can be derived from a 
single test administration. The issue be- 
' tweeajhe two may be of considerable 
th^^ffiS^importance. Practically, the 
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ooefficients yielded are not widely dif- 
ferent. 

We have . developed a table to sim= 
plify the computation if item variances 
and' die so-called "frue iteiri vajiances" 
Which are needed in this reliability 
formula. The table is entered with re- 
sponse couhts obtamed in t^e usual 
upper-low^ 27% iteirt analysis pro- 
cedure. You will note tha^a blank row 
watf left »at^e top of tBil tabla and at^ 
the.; left of it. This pen^its & clerk to 
enter dLfectly the number of correct re- 
sponses c6rresponding to a peTCBntagB 
of correct ^^pqnses^ for any given size= 
of upper and lower criterion' groups. 
When this has-been done it is possible 
to obtain the values of tot&l item vari- 
anee and "true" itent vatiance working 
directly from the response counts which, 
the scoring machines produce. Since we 
would be performing aft^fem analysis 
in any case* we have foiind this ap= 
' proach to "calculation* of test reliability 
much simpler "and more convenient 
than the split-halves technique. 

The pcobable error' of measurement 



of each tert is calculated primarily fto 
call the insb-uctor's attention to the 
magnitude of such, errors^ l^en with 
well constructed^ tests. Ristructors are 
cautioned, hoVvever, against regarding 
this probable error as a direct i^dex of 
the^ quality of the test. Some highly un- 
reliable test scores, which also show low 
.variability, turn up with very low 
■ ^ro^abre errors of rAeasurement, . 

V. CONCLUSiON v ^ p. 

' It' has Jbeen our purpose^ in this brief 
presentation^ to describe some of the 
facilities available »in the Examinations 
Service at the University of Iowa, to 
outlini some ©f the procedures lised in 
improviog examinations, to discuss 
some of the obstacles to more rapid 
improvement, and tO' describe specifi- 
cally our test analysis procedures^. In 
these, efforts to help cpllege teachers 
give bettbr tests, we ^ve gained a little 
ground^' but there is ampl^room for 
earnast and imaginative eiforts on our 
campus* and on others as %vell, for 
many years to come. 

.■■ ■ /■■ ■ / 
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ITEM ILLUSTRATING ^TEGOMES OF RELEVANaE 

....... ^ . - .- -.- - - ^ . ^ 



A, CONTONT DeT, 

"r , , "title* is a formal /^oM for a purely 
concepUial notion; I^ jo not know what 
it means and I 'question whethat'any- 
body does, except perhaps legal His- 
torians." Statement of ^ ^ 
' (t) Q[arles Clark » ; 
/(£) FmiX' Frankfurtef * * 

(3) Harry Chase ' ^ ^ 

♦(4) Learned Hand ^ 

'B. VOCABXJLARY ' 

'A security interest in. a chattels created 
by a bailment for ^e purpose of secur= 
ing the payment of a debt, is properly 
called 

(1) equitable chattel mortgage 

(2) deposit of title bonds 
*(3) pledge 

(4) equitable conditional sal© 

(5) conditional sale 

C. Fact - ^ 

The title of the mortgaged^ personal 
property is held in Iowa by 
♦ ( 1 ) the mortgagor 
(2) the mortgagee 

I .. ^ 

D. Generalization 

Probably the outstanding recent de- 
velopment in die area of the conditional 
sales contract is 

its gradual coalescence with the 
mortgage security devise 
(2) the development of the right to 
bar the dquity of redemption 



^3) the inequitable treatment jneted 
out to it the courts of equity 

(4) its total replacement of the 
chattel jnortgage . 

^E. Undebstand£no 

If a creditor ever got your advise on ^ 
loan arrange'&tents, ^ou might recom-. 
mend the taking of a deed absoliite in 
^form rather than a mortgage as security 
for a loan because (most , persuasive 
reason) 

"^(1) the debtor will have certain pro- 
cedural hurdles to overcome if he 
^ ^ comes in seeking to get the deed - 
declared a mortgage 

(2) the j creditor can move on the 
property on default 

(3) the creditor can sell, after de- 
fault, to a' third party free and 
clear and get the market value 

(4) by taking a deed absolute in 
* forms the creditor can ^bviate the 

necessity of foreclosure and thus 
eliminate the equity of redemp- 
tion 

F. Application 

A married to B, A alone mortgaged 
certain property. On As deathi B as- 
serted her right to her statutory share 
in the property. She claimed a one third 
interest in the realty. She can redeem 
by 

* ( 1 ) paying off the entire mortgage 

( 2 ) paying off her pro' rata share 

(3) having the court divide the prop- 
erty 
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Improvirig Evaluation of Educational .Outcomes 
A -at the ColTege' Level : ; 

r. I I. V f) T C II E N ' f 

THE EVALUATION DIVIDEND FOR THE INDIVIDUAL STUDENT 



Fortunately^ it won't be necessary to 
decide here whether a comprehensive 
pro-am of evaluation best serves the 
pmposes :ot tiie adrnfinisbration, the 
faculty or the studentSj for we all know 
that these a^a fruly interrelated. But in 
a sense it is the student who is the realN 
customer in most evaluation situations^ 
and if die program is to funetior^ fbr 
him in a useful and realistid way," we 
must assure him dividends in it. It is 
therefore periodically desirable to con- 
sider quite closely what the benefits are 
for the uidividual. This exercise can 
even be, if you wiU, a sort ot examina^ 
tion of conscience for the evaluator. 
Furthermore, because the evaluation 
process^ takes a heavy toU of student 
time and energy, it is only practical to 
insure that its outcomes be tangible to 
him, so that on the one hand he^ will 
accept the work burden more cheers 
fully" and on the other so that his moti- 
vation represent a maximum personal 
effort, For, after all^ this is an important 
aspect of obtaining reliability. 

In addition to these consideratibnSi 
there is that of multiplying student divi= 
dends by extending the evaluation re- 
sources so that students may also make 
ifse of them in that planning for which 
they are usually personally responsible^ 
as in their so-called "student activities/' 

Evaluation, as we utilize St at Penn- 
sylvania College for Women, ranges 
beyond the academic situation. We try 
to Help students to detarmine the objec- 



tives' of their CQ-eurricular ^ctivities^ 
fbr instance, and to study*the depee td 
which they are fulfilled by th|ir various 
groups. We/ work with tiiem to check 
unfounded rumors with facts and to de- 
termine what is the will and reaction of 
the majdrity in given situations. Evalu- 
ation thus serves an educational role in 
helping to promote a democratic spirit 
of inquiry and actfon. To this end, the 
questionnau'e type of inquiry has been 
utilized extensively. Structured as a 
process rather 'ithah as a mere fneasur- 
ing device, it has , had learning and 
other psychological effects ,of signifi- 
cance in improving the College pro- 
gram for the individual and her attitude 
towards it. In using the questionnaire, 
we make the overall assumptiQn ^af 
student participation in planning pays 
off in good dividends of rapport and 
learning. We have foiind that th,e ques- 
tionnaire method may be used to reveal 
to the student group, from time to timej 
the whys and wherefores of some of our 
educational devices and the reasons for 
planning in a certain way* so that stu- 
dents may gain a greater sympathy for 
, over-all objectives and some apprecia- 
tion of the difflculties that attend their 
achievement. 

We have certain long range objec- 
tives aiming at the improvement of 
GOurse examinations and thereby indi- 
rectly at the improvement of instruc- 
tion. Our office is also a "research" 
bureau in that we attempt within the 
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limitationi of a mod^t staff to aisembla 
data to answer or to clarify miscellana- 
, ous types of problems which arise from 
time to time. It is perhaps necessary to 
note that oun is a. small college of but 
several himdred students and tfiat wa 
do- hive'an axoellent faculty-student 
ratio. I am not so sure tiiat a larger 
institution could or should proceed in 
just die same way. But certainly diera 
would be a" way. All of our activiUes of 
evaluation are covered at a cost of ap- 
proximately $25.00 per student per 
year. This is the direct cost for person- 
; nel and supjilies. It does not cover such 
over-all items as maintenance^ which 
never get charged against th§ budget 
of any office, aor doesi it include the 
services of the faculty and stafif who 
may qp-operate from time to time on 
given projects or who co-operate con= 
h tinuously in the. general instructional 
and advisory programs of the College, 
In a situation where it is intended to 
give a fair^quota of time to follow-up 
of routine evaluation, there first has to 
be a r0sistance to a temptatibn to ad- 
minister too many tests^ questionnaireSi 
whatever, or to work with too many 
groups simultaneously, for a lot 6f busy 
work qan ensue that seems always to 
get in |he way of accomplishing the 
goals of; any one program, I have re- 
ferred to this as a tenjptation because 
while the preparation of tlie evaluation 
insttunient is certainly not an easy task, 
the adminis brat ion of evaluation ihstru- 
ments with their attendant scoring, list- 
ing, collating and first crude interpreta- 
^ tions is by far the easiest task of all for 
a person who is trained in those skills. 
This is even more true when published 
tests are utilized, But after the first ex- 
citement and interest of the more ap- 
parent results have been communicated 
to faculty and administration, interest 
can ebb dismally. The hardest task for 
the evaluator is to ke^ that inter^t 
J alive long enough to realize all the ob- 
jective of the activity apd to proceed 



radier like the durifty housewife who 
continues to itecure healthful sustenance 
for her family even from'^^the bones 
of the chicken Which she was able to 
serve so much more osteritatiously on 
the first day! An important portion of 
this susten^ce li^ in die Opportunity 
the evaluation program has to Work 
with students to devdop in Aem atti- 
tudes of self-appraisal and ielf-accept- 
ance and to plan multiple applications 
that will operate for student benefit. 

It may be ofjinterfst to note here \_ 
first the nature of the provisiens mSde 
^ to acquaint students with our services, 
Of course our students are conscious of 
our existence on campus from the day' 
they enter, since like other colleges, we 
test them for several days upon arrival. 
But at that time we make an effort to 
tell them why. A period of time is set 
aside for such orientation on the first 
mgrmng Just preceding the testing. We 
discuss each tost ancl why it has a place 
in the program, and specifically how it 
is to be used later for the benefit of the 
student. We also talk about the limita- 
tions of interpretation and the dangers 
of over^stressing test results. Within the - 
first semester there is another session 
with this group but in small group 
units. At that time we endeavor to 
acquaint tlie students with the more 
general objectives of the officej to help 
them establish some acceptance of 
them, and to enlist their interest. We 
al^o discuss course examinations, the 
pioblems that their instructors face in 
grading them fairly, thevreasons for ex- 
aminations, At a hiter time we discuss 
more specifically preparation for lam- 
inations. From then on as the necessity 
for any general examjnation program 
arises to fulfill administrative or other 
needs, students are carefully instructed 
in its purpose. They also learn in ad-* 
vance that each will receive ^ report 
" of the over-all results and a special re* 
port of her personal results which is 
usually issued in some profile form. 



i 



27 



TISTING PROBLEMS 



19 



Also, ^^knations ( are attadiad to the 
proU^ . that usually mthfy die peat 
majority of itudenti^ otiiera may drop 
in at A© office to talk^about these 're* 
siJtiuOr diseujs them with one of the 
dmm Of other faculty friend, 

^\ So much for om genoraUzed student 
' relations technique. Let us p6w con- 
aid^ some casei df indi\ddua)^^ students 
' and what tiiey receive in the way of 
benefits from tiie fact th^ attend a col- 
lege witii an evaluation program, I am 
■ h^e glaAdng over the generally ac- 
cepted beynefiti of better eumculum 
ftrough improved objeetiveSj better ex- 
aminations, , better t^ting^ conditions 
and peat^ efflcieney in handling and 
r^orting thejF results, for Ebel wiU 
have coVCTeA this aspect. My ^assign- 

:_\ ment here is to ^^eribe. some of the 
more tangible student benefltf. 

Our first case is just an ordinary stu- 
dent named Nancy who is not particu- 
larly outstanding as a good or poor stu- 
dent or problCTfi case/ Nancy was 
graduated from high school as a rnem- 
ber of the National Honor Society , but 
considering the fact that two-thirds of 
4e freshman class that j^ar had gradu- 
ated in at least the^ fifth ol* their 
classy, tiiis was not too signiflcant* 
During the freshipan week tests, Nancy 
distinguished herself in no outstanding 
way. With one or two exceptions not 
p^ticularly noted at that timei her 
scores .indicated that at leasts, twenty- 
five per cent of tiie class^'would surpass 
her later and that she would certainly 
in most instances always be better than 
another twenty-five per cent. Nancy 
fcaew that she could come in to sec me, 
or her faculty adviser, or her freshman 
dean about her test results, but she 
didn't feel any urge to do so until after 
hear third month in college. The impres- 
sion was distinctly getting through to 
her by tiiat time that sometiiing was 
. . ami^s^ -To make a long story= short, we 
had to get Nancy r^riented to the idea 

^ that tiie competitive picture was e^^ 



tirely differenti it was easier to do this 
by discussing her test profile against a 
backdyop of data for all other st^denti, 
This was done with an intention to urge 
her to put forth with a greater effort, 
nof to accept a mediocre rating* How= 
ever, it had to be done in a way to help 
Nancy become more selective about har 
efforts; and to give her .a new concept 
of the grade of C, which is*^ quite re- 
spectable grade in out College, Nancy's 

^ase^is representative of a common one 
among freshmen who for a while suff^ 
the rigors of increased competition and 
who could be hopelessly disturbed if 
they are not assisted to some bett^ 
acceptance of the change* It is prob- 
ably the most universal "common com- 
plaint" suffered in our freshman class, 
and all advisers are alerted to it 

Sometime a student benefits because 
a faculty member benefits first, I am 
tliinking of an instance where a student 

' fell into difiiculties in a certain required 
course in the social sciences. Esther had ; 
made one of the lowest grades in the 
class on any quiz the intructor saw fit 
to give^ she had turned in two poorly 
witten reports and failed to turn in a 
third; she was beginning to cut fre^ 
querTtly. The instmfator was a sensitive 
guy who last slee^^ver the occasional 
necessity of giving a failing grade. He 
stopped" in to talk about Esther. Her, 
odier grades were not remarkably good, 
but at least she wasn't reported as hav- 
ing undue difficulty in other ^ classes. 
The instructor suggested the possibility 
of moving her to some other section 

/ because he thought tliere might be 
something personal in the situation that 
was causing this block. So we looked up 
Esthers test records. Anywhere that 
she had encountered a test in the social * 
sciences she had managed to flub it. At 
entrance she had made one of the low- 
est grades in a test of reading and in- 
terpretcition in the social scienceii tibis . 
was noteworthy because results for all 
of her other tests, altogether about a 
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dpzen,. were quite good* At tfie end of 
th© fl^^toiui year, she had done just 
ai pdotly te a contemporary kff^i test 
— in f ^t^ ^ad turned flie paper in with- 
out Qoinpleting it, Her vocational inter- 
' est|t^t ihowad an extoemely low inter- 
^t s^re in the iodsl seryices area. It 
wasp't flia . ihs^ctor, that Esther re- 
jeotldi just his subjectl With .tiiis re- 
* assurance, the instiructor was better 
. able to cope^with Esther* I cant report 
diat she chose his field as a major, but 
she managed to pass the ^course re- 
spectably. The test records here had 
given some inkling of a problem this 
student had that might have ended 
sadly for her^ if someone had ^not 
bothered to^try to work it out. In this 
instance it is not Esther herself who is 
as typicil as the instructor. Students* 
problems are, frequeritly insfructbrs' 
problems, and our office can frequently 
helj^; 

Questions similar to these aris? daily 
in the office, in the eorridor, across the 
lunch table. 

AH of our test ^reports go out to 
faculty in list as weU as profile form. _ 
Many use diem to spot their problem 
situations in advance so that they may 
provide for them, What these provisions 
^l^ay be differ with the f acult>^ One 
.faculty person sets up tutoring arrange- 
ments for sWer' students; an English 
teacher may make a somewhat different 
assignment of term paper topics to in- 
dividuals; another may expect a great 
deal more of some than of others and 
ride diem when they don't produce; 
academic loads are adjusted; the Com- 
mittee on Aeademie Standing is more 
apt to take a sterner view of a failirig 
student of good potential than of a fail- 
ing student of lower poteritial, but on 
the other hand take more drastic final 
severance action in the latter instance 
dian in the fonner. 

Jeanninei ^se is typical of several 
other students who have recently made 
a similM' type of adjustment. Jeannine 



has an A,B. degree Witfi Honors from 
PCW, but Jeannine never graduated 
from high school We accepted her ^ 
after her junior year. She was unusually ' 
mature for her age with a high.serious- 
ness of purpose directed towards a 
career in music as a flutist. She was ad= 
mitte^ on that premise, her great deslrB^ 
to get on with her college education, 
her tamily s wiUingness to enter into 
the plan, chiefly because of the expense 
of her music education, and her tested 
background of excellent achievement in 
high school academic subject matter. ; 
She became a Mrs. Jeannine in her sen- 
ior year marrying a yoimg dental stu- 
dent. And she is now playing in a local 
symphony orchestra. Everything that 
Jeannine did was characterized by a 
singleness and earnestness of purpose. 
I talked with her frequently during her 
four years with us. She never regretted 
the move she had made, and her grati- 
tude ^ toward the program, that had^ 
made it possible for her was sincere. 
We have admitted a number of stu-- 
dents on diis plan, with no casualties. 

Many students benefit in the exemp- 
tion examination program. The exact 
operation of our program of screening 
tests aTid exemption examinations is 
complex and has been described else- 
where.^ Our studenfs after first being 
introduced to its opportunities are 
screened in various initial testSj and 
tlien selected ones are given exemption 
examinations in our so-called basic 
courses. If they successfully qualify 
they are excused from that course and 
enter immediately into the more ad- 
vanced work for which the course ordi- 
narily serves as a prerequisite. For. 
example, a student who qualified in 
freshman English enters Uie advanced 
writing course, or in modern society one 
of the more specialized social sciences 

' Lily Detchen, "A Program of Requipd 
Exemption Examinations,*' Jourml of 
Higher Education, (May, 1953), pp. 249^ 
254. 
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pto^mk Is ^ i^oditot is iM&gly 

of ac^^s^^ wor^ bat not achi^y 
t0qiv^M to do io. M^t do foUow our 
^ adid^ ^ Ao matter* Ha wiiqu€Qm 
^^^^ ^M^ QVff :^Qg^^ ii J^t OUT. jtudeali we 

ing axamuiatiODs, We baU^ve that tt^ 
program euti iAzou^ a lot of w^ta for 
diise stud^^^ itte^t^ ^€fir aca^ 
damiG afforbp and giv^ a suppor^g 
racogoition whl^ has r^ultad m faw^ 
dropouts Ml^^ pvtioiilar ^oup, 

* ' pvbm ttfrtutiet wa have emphaiiied 
to pyr aonstaruotiQn'work use of 
tho ^camination as a t^^lng and 
learning device than as a grading 

^ inibrument alone, I dp not wish to tay 

..^.^^^oQ-muob about-Aif^^iaie 

pam b^uia it ii the subject, of 
Dressel's pap^« But y/B have been ex- 
amining and tarying out various pro- 
to a^eve diese enda and will 

^ " e^^ io. Some of tiie devices 

whic^ have hBim employed have been 

% study sheets i^gh reiterate the objee- 
tivas of Ae course and ^ve tile studfnt 
swpla questloni for measyring her 
own progr^s, lists of broad study ques- 
tions aroimd whiA students can plot a 
study plan for diek final examinatiohs 
. wltih the knowledge that two or three 
of die ten or so quesdons which they 
are ^ven to prepare wiU be chosen as 
the actual final lamination ^say ques^ 
tions^ open-bookrexaminations taken in 
the dassroom, which it is hoped will 
encourage students to take and organ- 
ize better not^; open^book axamina- 
tions whi^ tile student is allowed to 

... . do outelde of the dassroom; ""illus- 
trate" examinations in science where 
tbe student Is ^ven a situation both in 
''wori" and in a "demonsfration" and 
it is ft^^t^ diat some additional 
iMfnin^ accmie as a r^ult of the dem- 

.--^-'^^ons^ations;, dtousilons m the clawooin 
of Aose conc^ti which a nriajority of 
students fail to achieve, ^ese weak 



spo^ ^vmg been located behind-the- 
scen^ throu^ the use of the IBM 
graphic itrafi ^unt^, a. 

At FGW a numb^ of uistmctional 
situations are built direcdy around t^ts 
or qu^tionni^^. In the Human De- 
vdo^ment and Bdiavior coursep Wfr^h^ 
man course^ Uie adminiab'ation of a 
vocational mt^est t^^s^v^ as a focal 
point" for die consid^ation of carets 
and the feature of temperament that 
must be taken into considmtion in 
loosing a jobi The lu'ger matter of 
valu^ in selecting carets is ako ex- 
plored. This class unit and Ae activiti^ 
of % Vocational Guidanee Week which , 
brinp repr^entative career counselors 
to campus are Mmbined into sometiiing 
more meaningful and helpful for the 

student, cl^room-instiuctor Hand 

the Siting counselors. In a s^or 

, p'oup a Life Goals -biventory is used at 
the outset of the course in Philosophy 
of Life as a sort of curtain raiser for die 
course* TTiis activity leads direefly into 
a^nsidCTation of purposes and values. 
Some students request copi^ to adniin- 
ister to their flancesj and th^e we i 
score for ^em also* 

As I indicated previously evaluation 
services are made available to students 
who wish such help wifli some imme* 
diate problm of student goyermnent 
or other student activity. In such in- 
stances^ the initial stimulation generally r 
comes from some faculty adviser who 
works with the group and who may 
make the sugg^tion to tiiem. One or 
two such projects have' been under^ 
taken annually* These ente^rises have 
been related to such problems as the 
elimination of faculty domination in in- 
terest groupSj the clearer defining of 
the roles of students as l^ders and as 
followers, the determination of the kind 
of religious pro-am that might best 
serve PCW studentSi a survey of int^- 

. est in the type of assembly pro-am- 
offered Ln a pMlcular year^ a survey 
of die attitudes of students toward 
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some specific aip^te of itudaat govern- 
Irient, analyiii of iamor opmion on tiie 
one hand md faculty opiirion on the 
otfier on tfie value of the s#nior tutorial 
^ ^e lUtimate aim m thrn^ projecti 
Is a sttucturing Aat will r^^t in betteT 
unSmtandinp for ^ose concemad 
through axunge of ideas and the 
assumption of rwpQ\isibility for leader- 
ship in planning. It is easiest to de- 
scribe the proc^i with an ^mmple. In 
one situation tiie student govarMient 
decided to evaluate the "clubs" (inter- 
est poups), class organizations, and 
the student-govamment association* The 
student planners of ttie evaluation de- 
veloped the following major sfrategyi 
(a) tlie utilization of a questionnaire 
technique to provide the college com- 
munity, boUi leaders and rank and file, 
with criteria by which to judge the 
effectiveness of student organizations 
and (b)*the provision of an opportunity 
for the student to consider in some 
systematic, way her confribution, ai an 
' individual, to student eo-oparative en- 
terprises. Because the ^questionnaire 
that the ^oup helped develop empha- 
sized that success of an endeavor 
r^ts as much with the general mem- 
bership as witii its leaders, the answer^ 
is^jtmi the questionnaire was expeeted 
toltprt more students to their resporisi- 
bilitiesi. That this questionnaire situa- 
tion was formulated as a process and 
not a mere data-collecting device is 
attested by the other activities which 
stemmed from it: 

1. In the planning stages several 
thodghtful sessions were con- 
ducted by a number of student 
leaders, who; In die course of 
analydng the criteria by which 
.the clubs should be judged, ac- 
quirad some needed, relevant un- 
deritandings themselves. 

2. The task of answering the ques- 
tionnaire actually was a procesa of 
self-analysis by each student of 



her attituda toward communis" 
endeavor and of her role as a con- 
structive coUegf citizen* 

3. Besides using the results for Aa 
immediate piirposes of making ad- 

^ justaente as suggested by student 
opinions a new group of student - 
leaders, entirely diflferent from 
those who worked on the ori^al 
planning of tiie questionnaire,-^ 
; was induct^ into its philosophy 
and use. This occurred at a lead- 
ership conference several months 
later, when the officers of organi- 
zations for the new year^recaived 
their" general gaining in offlcer- 
shipp The questionnaire results for ^ 
each organization furnished spe- 

i cific points for special review by 
the new officers of the organiza- 
tion. 

The use of tiie questionnaire achievas 
its maximum value when thosa^to whom 
the results aje of interest can ba given 
some share in,, its planning and when 
multiple interpretational uses are mada 
of the data. Incidentally, when inch 
questionnaires are given, usually Uia 
entire student body is polled. At such 
times a student committee prepare the 
materials, doing all of their, own mim^- 
graphing and stapling, administers the 
questionnaire, and tabulates the r^ 

suits/* , , 

In summary may I say that an evalu- 
ation progiam in a college situation 
must certainly include within its objec- 
tives that of scrutinizing its own confri- 
bution to the progress of the instkution 
and that unless this can be done fairly 
concretely in terms of benefits to the 
individual student, there is no real as-, 
surance that its activities have much 
meaning. 

^ Lily Detchon, "Instructional Values As- r 
soeiated with the Use of QueiUormalrei, 
Sdwol Reoi&w, (November l9o3), pp^ 
481-486. 
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EVALUATION AS INSTRUCTION 



As A REFORAfflD (or possibly tmBgade, 
dep^ding on tb@ Gaining of ob- 
smv^) mathmatioian wid itatistiaian; 

J I retain suflBdent respect for the eon= 
QBpt of intepatioD; to winQf at the logie 

' and indelicate tuage of ^e tmn whi^ 
- baa beoome cwtot in tbe field of edu^ 
cation. Yetj ^^erienc^ of the past teto. 
years as a eounjel^^ organize attd di» 
r^tor of a counseling center, as a test 

^ constructor, organize and head of a 
staff of ^camin^, r^earchers and eval- 
uators, as a director of an evaluation 
project in genial education, and as a 
speaker and c^nsid^t on general edu- 
cation have broii^t me to a state of 
intoospection and to an attempt to or- 
ganize my varied ^cperiences and reac- 
tions to th^ which^ can p^haps best 
be qonveyed by the word "inte^ation," 
I should like to share with ^ou an 
initial attempt at mtegrating or organ- 
izing my Uiou^ts in measurement and 
evaluation as diey rdate to the educa^ 
tiQnal procesSi In so doing I make no 
claim to originality nor even to a com- 
plete and coherent ^Uating of the 
ideas of oth^s on subject. 

In^^sta^y I have felt, and I loiow 
that others have shared with me the 
feeUng, tiiat a wide and widening gulf 
separate the |premost tiiinkhig in 
measurement from the reality of meas- 
urgent and evaluation as cairied on in 
ckasroom» ~I -Ihilly recognize tiie 
neM to push ahead on ttie tedinical 
and Aaoreti^l front but ^ncmi^ as I 



am with the implications of m^ure- 
ment for thfe improvement of ie edu- 
cational proce^g, I would sugg^t 4at 
^^tiiere is an equally urgent need for re- 
lating some of our Ainldng^bout meas- 
urement to these mattersClt is for this 
reason tiiat I have ^ehoswi ^ thr topic ^ 
Evaluation as Insbruc 

Perhaps somQ sli^t. residue of niy^u 
maAematical Uaiiimg prdmpte me to 
start witii a set of assumptions. After 
numerous attempts 'at produping suc^ a 
set I have arrived at ten assumptions^ 
paired in such a way as to suggest what 
seem to me to be parallel element of 
evaluation and ' instruction* Indeed, 
there is more than a sugg^tion tibat 
good instruction is simply continue 
evaluation. The asiumptionSj whic^ are 
in part also definitions and principle 
of learnings are: 

1. InsfaTiction is effective as it leads r 
' to desired changes in students* 

2. Evaluation is effective as it pro- 
vide evidence of the extent of Ae 
changes in students* 

3* New behavior patterns ^e b^t 
learned by students when die in- 

» a3equaey of present heavier is 
understood and tiie significance of 
^e new behavior patterns th^eby 
made clear. 

4, Evaluation is most conducive to 
learning when it provide for and 
enOTiu'ag^ self-evaluation, 

5. New behavior pattmii can be 
more efflciently developed by 
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■ /^ 'taaciM who tt© ^ting 

V toMvlcff pattern rf in^Wdual 

6. Equation is wndudve to good 
abfuetioii wh«a it r^aal^ major 
types of taa^vat© behavior and 

T. LeOTing ^ ^oourajged by prob- 
1^ Mid activito ^hl^ requke 
Uiou^t wad^or i^tion by each 
indl\ddiial student* 
8* Evaluation ^ most signfflcant in 
learning when it parmite and en- 
courage ^e ^ereise of ^dividual 
initiative, i_ • 

9. Aetiviti^ which provide the basis 
for the t^chmg and IpOTihig ol 
^edfled bAa^dor are also tiie 
' most suitable aativiti^ for evok^ 
irig^^ affd waluatoig Ae ad^uacy 
of ^at behavior, 
10* Artvities or ec^cises developed 
for &e purpose of evaluating 
'Specaed ^ehayipr are also useful 
for tiie teachirig arid leaming of 
that behavior. 
' As one immediate result of these as- 
sumptions we oonolude diat: t 

Evaluation does not differ from in- 
' sbiiction m i^mposes, in methods, or 
' in materids and can be differentiated 
from instruction only when the pri- 
mary pu^ose is diat of passing judg- 
ment on the achievement of a stu- 
dent at tiie close of a period of 
insbiiclion. ' . . 
I should lie now to examine with 
you what I believe to be some of the 
major implications of th^e assumptions. 

First, in regard to teaching it ap- 
pears ^me tiiat die following state- 
ments may be readily inferred from the 

preceding assumptions: 

1/ Classroom practices which are re- 

stticted to textual or teacher pr^- 
entation of knowledge and Ihe 
testing of 4e extint of recall of 
tawedge are unworthy ^of 
tihe name tisfructioni 
, Su^ practiQes, regMdless gf stated 



objeetiv^i m^e knowledge &e 
sole dbjeetive of insfruetion. 
Learnhig is umeal^tic in tot 
what is learned is ^Voreed from 
reality. The teachCT covers mn- 
^ tent but does not insttuot students* 
The majority of students remain 
completely passive, and work only 
to memorise what the teacher 

CTnphasiz^. ^ 
2, Good instruction will be eoncemed ^ 

^ l^s with what die teacher is do^ 
ing or wants to do and more vdth 
what die student is doing. 
It is readily accepted tiiat die su- 
pervisor who devotes most of h^ 
time to planning his owi work 
will not last longi he must accept 
as his major r^onsibility the 
planning of tiie work of ^ch of | 
^ his workers and die assisting of. 
them when difficulties arise. The 
analogy is not amiss in considering 
the task of the insmictor. Learn- 
ing is an individual phenomenon 
and results largely from the effor^ 
and activities of die individuaL 
Good insfruction involves careful 
- planning of specific tasks wi^ - 
definite purposes which can be 
undertaken and completed , by 
,^aeh student. It is' not that which 
the insfructor do^ that counts; it 
is what he stimulates die student 
hLmself to do that yields the most 
significant educational results* 
3, As new materials and skills are 
presented^ the insteuctor v^m 
simultaneously assign tasks whidi 
* require applications of tii^e by 
the student, thereby providing 
both to the teacher ^d the stu- 
dent concrete evidence of die 
utility and of the mastery of tiiose 
materials and skills. 
This conclusion is simply an ex- 
tension of die preceding one sug- 
gestihg that ihe assigned tasks not 
be repetitive or copywork in na- 
ture but require relation of new 
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" J mata^s and ikilli to old ones 
, wad the application of both to tii© 
adiiovement oft a deeper insight 

. toto old or hew problemi, 
4. For motivation of laming and for 
continual evaluation of tfie effec- 
tiveneis of initruetion the instirue- 
tor will dieek dmost itoUy for 

, evidence of change in stiidehta 
and wHl seA to make diese 
ofaahges as* e^dent and urider- 
standablf to them^ai to hlmielf. 
A wmmon complaint of students 
is that ttiey* receive tes tiiwi per- 
fect gradei and have no faiowU 
edge of Ae imperfection* In com- 
< position and speech work a similar 
complaint by students that tfiey 
see no improvement over die pe- 
\ riod of time in which they take 
die course* I am convinced^ tl^t 
diere js real ground f or^bodi com- 
plainti in uiat ins^ctor^ them- 
/ selves ^e not commonly aware of 

changes in individual students 
and certainly make little attempt 
to provide cono'ete e^ddence of 
change to the individual himself. 
The irritation evoked in mapy 
teacheri by students who want to 
^ know precisely what is wrong 
with Uieif work is perhaps less a 
resentaf nt that a judgment is be- 
ing questioned than it is in the 
reahzation diat tha^ judgment is 
highly falhble and hardly defensi- 
ble, 

5*^To encourage integration ar^ 
toansfer^ die ins^ctor will seek 
for tasks from or impinging on 
other fields of knowledge which 
do or can utilize materials and 
skills prsently in focus in the 
course. ^ 

The association of ideas, concepts, 
and principles from one field with 
those in anodier, or with cun'ent 
problems is one mark of a well- 
educated person but it is very 
hkely to be frowned upon in the 



classroom in fear Uiat die associa-^. 
tions made will be superflcial The 
instructor Etoself, ttained in one 
iBeldt h^itates to make reference 
to others, and die student who 
attempts it courts ridicule or rep- 
rimand. Yet life is not depart- 
napitalized Uka alleges and the 
. development of indi^duals of dfie 
ability to integrate knowledge and 
experience and to ttansfer it rie- 
quires practice in it* 
I am aware that many teachers pres- 
ently regarded by colleagues as highly 
^mpetent and even outstanding would 
hotly debate or as coldly ignore some 
of diese points either on die grouiids of 
irrationality or impracticability, but I 
am much more troubled .by the fact 
that many teachers who assent to these 
views depart so far from them. In their 
own practice* Yet^ since ouj/prim^ 
cdncern here today is widi measure- 
ment, I must ignore the many issues 
raised by diese remarka about teaching * 
in order to deal with odier more im- ; 
mediately relevant matters. 

In regard t> evaluation I find my 
assumptipns ajove leadmg, to the fol- 
lowing conclusions i 

1. Testing for padlng shoidd be, 
relegated to a minor role in in- 
smiction and possibly even as- 
signed to an independent evalua- 
tion agency* 

2. Testing for knowledge should be 
supplemented and even in part 
replaced by broad, pervasive, and 
continuing evaluatibn or assess" - = 
ment which become the major 
part of insbiiction and therefore 
indistinguishable from it 

3. Testing practices which involve 
. ^ comparison widi status norms for 

liuperflcially similar groups should 
be replaced by practices which 

.^emphasize comparison with maxi- 
inum gains made by students of 
similar back^ound and ability in 
regard to the same objectives. 
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* . 4. Condftn ^^jreltabflity and v^id- 
it^ as itetotici conaepts Aarac- 
^ torang a^^uaOTn initfumante or 
piocedw^ ihould be replaced ^by 
cbncOTi with the p^^anpncy and 
, rdevancy of leamuig as i^nce^tl 
charactering tfie quality of in- 
) " ; . struotidnt 

5* PsychofiieWoi — at least as pr^ 
tieed by Aosa'concOTifd with in- 
stfuctibn ^d widi learning— 
should become somewhat leas 
Qonc^ned vdA ""radier exoterio 
ni.a^ematical ^inv^tigations of 
measuranent ^tiiwy and iriora 
" conoemed wiQi tiie proposing of 

models aijd hypotheses directly 
useful and testable in elassroom 
y , and life situations* 

' Thefe conclusions indicate tfie need for 

sortie dianges in our evaluation and 
measurement practices, Each of them 
is sufflciendy dgniflcant to. call for some 
* '^discussion. 

I have 'suggested that testing for 
' ' grading should be relegated to a minor 
role in instifiietif n. During^ a period of 
visiting c1|i|ses in many different col- 
» leg^ last ^ppt^ and spring I foimd 

* practically cases in which there was 
any indication that testing or evaluation 
was carried on for any purpose other 
than grading. Frequently^ testing ses- 
sions are reduced to one or"^two per 
t quarter in order that tiiere be more 
. time for covering the ^ound. Hali- 
' '' jokinglyi but none-the-less ierlously^ in- 
structors stiU hold the threat of pres- 
ence in an examination as a reason for 
paying attention arid learning of a par- 
ticular point. Teste and examinations 
are stiU eatastropfiic events which are 
greeted by the students with disniay, 
for testing for jading is inevitably a 
direat to studente and not a procedure 
, which is either conducive to motivation 
or to learning. With rare -exceptions^! 
doubt that an ttismic can have really 
close association with his students and 
get a ttiia picture of what Aey know 



or what Aey^w'e thinfting about while 
he brandish^ Ae weapon of a grade. 
If you doubt tiiiSj may I raise tfie. 
analogous question as to how fre- ' 
quenfly an instructor gives vent to his 
ajctual feeliftgs and reactions about an 
idea proposed to him or to die stalF by 
hii dean* Autiiority in the Sorm of tiie 
power to reward or to punish inevitably 
mbdifi^ the overt thinking, expression, 
and action of diose to Whom that power 
is applicable, I must hasten to add (hat 
experience in working with a staff in 
which all grading was taken from their 
hands indicates that sudi ^Vorcement 
cannot be successful unless the staff 
itself accepts the principle evoked in : 
tli^e remarks, I hope, however^ and 
I believe that it is not circular reason- 
ing on tny part to say that those teach- 
ers who have given most evidence of 
fully understanding general education 
objectives and have evidenced most in^ 
terest in the needs of individual stu- 
dents havd been those who have radier 
readily accepted the principle of di- 
vorcing grading from instruction. 

So far my remarks have applied per- 
haps more to teaching than they have 
to evaluation but they have implica- 
tions for evaluators, So long as teachers 
are primarily interested % the use of 
tests which enable them to assign a 
grade on content materials, teachers and 
test agencies will of necessity spend 
most of their time on the making of 
achievement tests which emphasize 
knowledge of specifle facts. There is 
more than an indication, however, in 
the light of my experience in the Co- 
operative Study of Evaluation in Gen- 
eral Education that many teaches only 
await the availability of insfruments 
and techniques which, will provide evi- 
dence of the development in students 
with regard to other types of objective 
to make extensive use of such evalua- 
tion materials. 

To illustrate the insidious effects of 
the grading practiceSj I want to remark 
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briefly upon some of oiir e^eriencea 
with insfrumenti dealing with affective 
objeedves/ One fnifrument developed ^ 
in our Cooperative Evaluation Study 
was^ an Inventory of Beliefs/ which 
seeks to get at certain attitudinal ele- 
ments which ^e related ttf general edup 
cation. The stotsnente in rtie inventory 
are stereotypes or prejudice ^commonly 
and freely expressed by individuals. 
When studente are asked to react to 
this Inventory^ they must be assured 
very definitely that tfie results will be 
in no way used for grading. There have 
been cases in whidi students have in- 
dicated that they have not given an 
honest -answer to the statements be- 
cause they feared that an individual 
instructor might look at the results and 
be likely to recall and hold them against 
him at a later date. Similarly an in- 
ventory of attitudes toward the human- 
ities and the people working in that 
area was made up of statements actu- 
ally extracteS froin students in informal 
situations or from essays in which they 
were writing on other topics. Flaced, 
however^ in tiie context of other similar 
items of tiii^ types we found that the 
humanities attitude inventory gave very 
little in the way of ' a range of scores, 
simply because students realizing now 
what the inventory had been developed 
to do, gave what they readily perceived 
to be the approved response to the 
items. Their inner feelings on the mat- 
, ter were concealed partly ^ at least, be= 
cause they did not titist the instructor 
to deal fairly with them if he knew how 
they really reacted. 

It has been said that the grade 4e= 
termines what the students work for. 
I think it may equally be said that the 
possibUity of grading a student' deter- 
mines to a considerable extent what the 
ins^ctor considers important. Cer- 
tainly, teachers are incUned to ignore 
objectives for which they have no ready 
or defei^ible means of ^evaluation and 
grading. I also feel quite certain^ and 



I do not' know whether, it is fortunate 
or unfortunate, that student generally 
speaking are less concerned about 
grades than are their teachers. 1 feel 
that there is only a limited set of ob- 
jectives upon which we dare to grade 
individuals and tiiat so long as a grade 
holds the most prominent position in 
our tiiinking 1n tiie classroom it will re- 
strict both our insfruction and evalua- 
tion to thi$^ limited set of objectives. 

My second conclusion regarding 
evaluation was that testing for knowl- 
edge should be supplemented and in 
part replaced by a broader evaluation. 
, As this is done; evaluation becomes a 
m ajor part of instruction and ultimately 
indistinguishable from it. I should hke' 
to illustoate by a number of concrete 
exarnples what I have in mind by say- 
ihgdhat insti-uction and nevaluatioft can 
bec&ie ohe and the same thing. The 
AngOT«Tro}^er SeIf=Scorer, sold by Sci- 
enje Research AssociateSj provides a 
means whereby a student taking a test 
continues to select answers until he ob- 
tains the right answer. If the answer 
that he first chooses is incorrect, in 
purichitig tlie answer sheet he finds 
only a white space revealed; He con- 
tinues punching until a red mark shows 
up ^ under the hole punched out and 
then he is assured that the correct 
answer has been obtained. I have ree- 
ommendcd to numerous teachers that 
this Self-Scorer be used. While some- 
what inconvenient to handle^ everyone 
who has worked with it has found that 
very favorable student repction is 
^ created, A technique frequently used 
has been to have a small number of 
questions, perhaps ten or a dozen, cov- 
ering the major points of an assignment 
to^ be discussed for the day. The stu- 
dents answer these questions on the 
Self-Scorer. Those who really know the 
answers are assured that they are cor- 
rect. The larger number who nfay have 
been, when they came to elasi, reason- 
ably sure they knew something about 
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ihm aiiignaaient mi ^$^£nd ttiat< 
must ^0 or motm s^^Uom 
^ fore flriding' :^9 ri^t imiwir are moti^^ . 
vattd to &ki out why» Tliose uidi\^d- 
who had to pun^ aQ four mi^m 
brfore getting ft© ri^ one ar© c»n- 
cfmek about it, Thmm fa litU© diffiaulty 

t^^ ih gHttag a tram ^ouMion wb^ tihe 
test itoni have been qpmpletadi An in- 
t^^tlng point in couneotion with the 
Salf^ScQrer ii ^t iinoe Uie ttud^fc has 
pimdied a hole in the answer sheet he 
may retain tiie insw^ sheet in bont 
of him Uirou^out a disousaion with no 
conoem on tibe pvt of ^^ tmchm tibat 
any modjfiQation will be made in the 
r reaulti. ITiere has been at least^ one 
inv^tigation which would indicate that 
the use of the AngeU^Troyer Self-Scoper 
results in ff^t^ retention on the part 
of ffie student and Ihe suggestion in the 
same investig^on that the Self-Scorer 
followed by a discussion of the results 
would restdt in still ^eater. retention, 
One of Uie major valy^ of the device 
is tiiat each student is forced to go on 
record with regard to each item. The 
usual tendency in most classes is for 
eithiftf the instructor or a few students 
to take the lead tii answering questions 
and discussing points* The reticent in- 
dividual easily arrives at the decision 
that die answ^ which is flnally indi- 
cated as H^t is the one he felt was 
ri^t all the time, He just does not real- 
^ ize that if he.had had to put himself on 
record at the beginning of the discus- 
sion he vv^uld have marked an incorrect 
answer and he has not consciously ana- 
lyzed the r^sons for his ineoirect posi- 
tion. There is real value in having in- 
dividualj. realize just how much or how 
little they laiow and thereby y ve them 
some incentive to do sometiiing about 
it 1 

iy a second sample of die coinci- 
dence of ev^uation snd ins^ction I 
^ should lUce to refCT to the Theme Aniply- 

r sii Hwidbook deVeloJed m our Cooper- 
ative Evaluation Study. This Handbook 



ori^nally developed out of a coneernr 
on the part of our Coxj^imimications 
tfeachers tiiat thwe he some bhprove- 
ment in thttne readingi A large nuni- 
b^ of themes wei^ collated fronh all 
of the colleges involved in the pro|ect 
and teachecs undertook to agree on a 
^ding for tii«n* They soon fou^d 
diat they were reading tiiemes from 
different viewpoints aiwi wiA different 
pets of values in mind, it bec^e clear 
that they had to formulate some kind 
of a itatement of intent or purpose b^ 
fore they could move to any a^eemmtv 
GraduaUy tiiey found diat many of 
their difference arose out of the fact 
that die reading was rather superficial 
in many casesj tfiat one individual was 
emphasizmg one aspect of a theme and 
another individual another aspect. Too 
frequently all readers were missing die 
^point of examining what the individual 
student was really frying to communi- 
cate — a viewpoint which' requires con- 
sbructive suggestions as to how the in^ 
.tent of the student might better have 
been accomplished, ^ this became the^ 
guiding . p\irpose in dieme reading it 
was found that much more agreement 
could be reached, although die task' 
of reading a thKne in this fashion be= 
came a very arduous one. The reader 
must determine just ^hat it is the stu- 
dent is ^ing to accomplish in the first 
place and having reached this judg- 
ment he must then re-examine all 
phases of the dieme in such a way as to 
be able to make concrete suggestions 
to this student as to how die ^CTfie 
might be improved. This becomes less 
a task of assigning a ^^de to a student 
and much more a^ task of helping die 
student to evaluate his own effort* 
Firiallyi a rather large. , number ^f 
themes were thus evaluated and put 
together in a theme analysis handbook* 
The first purpose which the committee 
had in mind in using this was that it 
would be a device whereby ti^^ers of 
Communications coiJd work togetiier 



oh a let of and imdi ioroe com^ 

: mon^ty in Ainldng, A gecond 
usage envisage ^as that givteg it to 
nl^r Wbuctoi^ on a staff m a basis for 
halping Uiem davalop the skills of 
/ Aama raading. Soma taa^m, how- 
'\ ev0r- ioi&d that to »sa i^me . Aames 
Nvlffi ^ opaqua proji^tor providad an 
int^^ting and ix^teotiva akparienca 
for Qia itudents« Tha itudents could fare 
ask^ to maka &a same kind of critic 
cism of tfie diemg ar dia taadiars had 
engaged in* AftCT tiiis criticism had 
baan randarad it wasffdfan possible lo 
dirdW on Aa s^rean the analyiis made 
by th© Communications taachars and 
the studants could daen, point by pointy 
coinpara their own reactions with those 
of the committee. In a sense, this is 
■ only a stib^tituta for making ^ugh an 
analysis for each individuaL It is quit© 
obviouSi howaveTj that a teacher can- 
not take tha time to make the detailed 
sort ,of analyiiSs. diat was involved for 
ey^ thama of tvery student There is 
some indicatioh in a subjective w^y 
diat a' student who sees a theme ana^ 
lyzed in detail after haidng undertaken 
himself to make such an analysis ob- 
tains a better idea of die problems of 
communication than he had before. 
Thus, a devic^ ori^nally ii\tended as- 
* an valuation device has been found to 
have toamendous insfructional value. 
The Thame Analysis Handbook may be 
regarded as tiie placing of die evalua- 
tion of themes on a concrete objective 
level whereby the evaluation has obvi- 
''^lous significance to students in indicate 
ling die quaUties of good witing, 
\ The Test of Critical Analysis and 
Jud^ent in the Kumanitias is still an- 
oth|^ example of the same coincidence 
of insbruction and evaluation. Teachers 
of Aa humanities indicated 4 concern 
diat student faced with a work of art 
be able to arrive it some judgment of 
it based on an application of factSj 
= - principles, %nd p^sonal reaations, They 
found that when students were simply 



asked to react to a picture or to a piece 
of music, that practically nodilng was 
obtained, An attempt to evaluate die 
itudents' ability in critical analysis and 
jud^ent failed simply because most 
''Students in most classes had been given 
no opportunity to develop such facili- 
ties and tharefora w^^unable to ax- 
Jhibit them. In most cases it was obvious 
that the t^sk of analyzing and Judging 
a work of art was something rath^ new 
to the student "knd diat he was even 
be^^ldered by the request. TTiis being 
jtiie it might seem that we should con- 
eenfrata on teaching. On the ' other 
hand, askiiig just what kind ot an analy- 
sis and judgment or just how a judg- 
ment might be' arrived at for a particu- 
1^ work* of art holds promise of 
clarifying ways of insbiieting students 
irt order to develop Uiese abilities, As 
agreement was reached on a number 
of pointe and a seri^ of^ suggestions set 
up for students as to things fe| which 
they might react in analydng mS work 
prior to making an integrated judgment. 
Whfn this was presented to studente, 
it was found that sometiiing mora 
tangible in the way of ^a response was 
obtained ^om studente. However, the 
vast majority of^ students stiU betrayed; 
a lack of familiarity with diis objective. ^ 
The structured essay format adopted 
for the critical analysis and jud^ent 
in the humanitiel^ is probably most 
valuable? in deahr^ widi studente who 
have had little e^^erience in this type 
of work. In othen^prds, the instrument 
can be used in making assignments or 
it can be used as a guide in class reac- 
tions 4o a work of art. Ultimately, how^ 
ever, when a student has had enou^ 
practice it' would no longer be nec^- 
sary to put before him the specific 
points in th© sh'jucture of tha critical* 
analysis and Judgment, tfst- He^ would 
already have afrived at that sjtucture 
or a similar one based "upon nis^ own 
ei^erience to which he coum r^ort 
when faced with such a taAf The de- 
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i vim b stiU a tM md ba used ai 
/ siidi, hut if anyftiQg iU value as a 
t^ching devic© ^c^ds its Slue as an 
ii evaluation devjce. 

I ibould l£k© to use on© more 
ampW to iBuitmte tfi© combination of 
' ^ inibruotion and evaluation* Hie icience 
^mMm of ow proj^ b^me inter- 
to the use of cment ^qience tea- 
teriaU begause of ^ feeling that the 
fu^ftf contact of many stjudents taking 
ody one general edu«ttion science 
OQurie would be Iwgely wiA popular 
and semi-popular science articlei. If 
students develop an int^^ in reading 
Ais matmal and |ome skill in applying 
their loiowledge of scianea and their 
iimlfumiirljTir of tfia sci^tific attitude 
toward these materials, if might ^b? reaf^ j 
. . sonably^eetfd thats ^ » , 

1. They would continue some con- 

iact with science. 
2* They would continue to increaie 
tiieir knowledge of the area by 
tiiis Constant reading, ^ 
Tha task of evaluation resulted in an 
extensive search of science articles to 
see what kinds of materials might be 
appropriate for avaluating students* 
ability in science reading. It soon be^ 
came evident that some articles wetp 
^ much more suitable than others. It . 
also becama clear that having an articlS^ 
Was not the same thing as kno^^g; 
exactly what one would do. who rem 
it CTitically and understandingly. It be- 
came nacessaryj ther*eforei to build up 
a 1^ ©6 behaviors which might char- 
acterize tfie critical and understanding 
reader of science materials, These in 
turn were formulated into both essay 
^hd objective t^te. Use of di^e in the 
<• cl^srooms indicated diat students did 
2^*^ ^ iipf piffbnn such tasks and 

r N\ tiii^^^vsm^ indicrtyie'of the fact that in 
Itnost dlkssrdpmr it was the first time 
*I 1^ V^fft studerii had been asked to make 
, : . , -analysis ;^Qf any reading ma- 

• s teritit'Aj/^^r^ult boUi objective test 
"^^0^^'''and toay. questions raising 



issuei about cuffent articles and ex- 
tacte from articles, were found by 
numerous teachers to be a ehallengfng 
,^e of activity for tiie classroom. Such 
inaterials evoked much interest and dis- 
cussion on Ae part of students^ andj 
often for tije first timei caused each in- 
dividud shident to attempt for himlelf 
to do ^mething widi tiie material , in 
frortt of him. Textbooks and. teaches 
who pr^ent botii problems and solu- 
tions to sbidents rkrely challenge them 
^because it is an exceptional student who 
can undertake to disaffrea with a solu- 
tion proposed By a "teacher. Problems, 
with Solutions not given, involve a . 
much less threatening situation in 
which "each student can propose his 
Gwrf solution and argue for it up to die* 
point wjiera he is convinced ^that therfe' 
are better solutions than Sis. Such use 
of wrent science material in the class- 
room results in a coincidence of evalua- 
tion and insfriictional practice. Even 
more significant, the type of behavior 
envisioned by a group of science 
teachers for their students at a peripd^ 
later in life— that is the Reading 
science materials becomel > a part of S 
* daily classroom experien^^r Surely the ' 
problem of transfer must^be much less 
. complicated for the student who has 
^ been asked and even force<J^ read 
^ current science materials. It is far more 
likely^ ^ tiiat ' h| will continue that be- 
havi^. ti^n if is that a student with no 
such expefienSe ^11 initiate it just be- 
cause hi^ has' leamed a little more sci- 
ence. » 

The third conclusion which I have 
suggested as applying to evaluation and 
acfiing out of my earlier assumptions is 
tKat;*th0r^e should be a de-emphasis of 
natiQual and regional norms. To illus- 
trate \|hat I have in mjnd lei ma dis- 
cusi;.fbr |ust a moment the nature of 
some of t^e^ Cains which we found on 
tha Test ot ^ftical ranking in Social 
Science. In sonia schools we found over 
a year's time a gain of one or two 
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pdinte on the av^ag©,.a gata swcdy 
. tnore thm practice efffot dette- 
mined foir .tfiat test. In o^ct sefcbolj we 
found a m^n gain of si high as tan 
ior twalv^ poBit^^ atefiost aTl^ casai 
, found tiiat ttie students who were lower 
t'at bagirining made much\ larg^ 
gains than tije itudente who had a high 
score on die test at the beginning. Par< 
1 tieularly it w^^ true that a student 
whose score on tihe test of OTtical link- 
ing was lower relativaly^ than his in- 
telligence level tended to^ make lajge 
gains. Yet there wer^ plac^ where tiie 
i most able students mad^ larger /^ains 
; than the least able students in other 
schools* There were plapef in which the 
dffierence in the gains between the 
most able and the most Inept stiidfents= 
..was puch less ^an, w&s\the pase in 
dthef schools. Such variatiGns are, un- 
doubtedly associated with variations in 
the educational pro-ams although it 
was not alway^^ossible to identify 
j them* jfn gener^'rf ha large gains were 
made wh^re more oy art attentiph . was 
givari to the objective. In aqcord jwith 
the pr.evailiiig trends' We : threw together 
' the data from tfiree major^cla^sifications 
of schools bafed on the abjlity of the 
students admitted and mide^ up what 
might be. called norms. The general 
implication of these is that a school in 
which the students nj^ke a gain of 
■ roughly four points oyer a year's time 
'.fan the; test of Critical Thinking in So- 
/dial Science is doing an jav^rage Job., 
[ The difficulty wjth this is tiiat an aver- 
- age job is^^spKiking fraiMly^a dam 
' poor' showing, Most So^ml ' Science 
i classes f in general educat^bh are still 
^heavily^ content eenteredj w qst-students 
•are g^ven no ogportunity tc practice or 
idlevelop critical .ttrinking, pnd only in a 
few scj^pols wit^very outstandiiftg gains 
'lean on^ find in the instructionaK pnic^ 
ki^ce .anything whicli is liidicativlfpf real 
A.attieAtion 'to the objective* EorJfchoQls 
t:,artd for teachers to mdopt' as their 
,;^$tandard :the mean gain found p a set 



' of, norms p pteliantly derived is staply ;\ 
to ^vp blessing to mediocrity, oi wdrie. 
The- i^f^l ^^andMli-^Wch ^ 

^ SQught jiftAe largest; ^ain^^^^ stu- 

. drtts^f 'tHe jania ability level and gen- 
eri^: Sabk^ouhd. It would tiien be 

; ■ nec^^l to. find out just what Were the 
ch^fteteriitfcs of the program which 
aithieved*^ those gains atid to adopt as 
many; charact^isycs as are appropriate 
or evei% to improve upori diem in ord^ 
to\gain'still mora. Another point which 
I havf ^hpted is that most norms com- 

' ;mbnly hrfi^ only given in terms o| a 
ctosiBcation of types of ^lools and 

: oeeasionaily Hn taftns of intelligence 
leVe1=^0i3r finding that low. ability ptu- 
■ dents, with law, pre-^t status on the 
Sociafe Science/ CWtic^^liinking Test^ 

. v.are ;unlikey^' .td make very large , gains 
vis' only^ cqnifhqn sens'd. jrhe faqlk Uiatf 

/^ students with aVera^e dr higli scores on 

; tfaCritical Thiinkfcg T^t in Soda! Sei- , 

enee, who haVa Jbwi aiSademle ; ability, 
: make almost nq gain lyhidhiis also very 
rerfsbnable/'^t with :the'\Usu'aI no 
ptpvided, a feapher who* is- aware of 
some of th^se iSituatiqhp -W^Id find it 
very difflcult ,or injpossibje .to obtain 
any information 6n,,what ^buld be a 
reasonably, espected 'fmprovement on^. 

' the part of a given styderit. Perhaps we 
should take a leiif 'from flie practice of 
photographerSit both amateur and pro* 
f^ssional.^ Those bf you who are inter- 
ested in the flcld kriow^tliat it is cus- 
tomary when displaying a photo^aph 
to provide a detailed descripflon of die 
cjnditions under which it was taken, 
th& kind of exposure, lens, etc,, used. 
Like%vi^ej- it might be appropriate in 
educational circleSi when tests are given 
and^ gaiUs are recorded, to describe in 
^some detail the essential characteristics 

. of Hitf educational experience which 
was provided. In closing niy remarks^ 
on norms I think I should make it -clear 
tlmt I do not feel that norms are neqes^ . 
sarily jsvil in themselves. There , are 
some ^^tfliled,yalue&^ s^ed ^by them. 
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On oAer hand, tfia prevalence of 
national and regional norms emphasiz- 
ing mediocrity bften leads teachers to 
regard them as the only kind of standard 
which Js significant in judging test re- 
sults. I suggest that it is a major obli- 
^tion of everyone in Ae measurement 
and^evaiuation field to see ttiat the aU 
most exclusive use' of status^orms for 
superfldally similar poups '{6 supple- 
mented or even replaced by practices 
emphasizing comparisons of the gain 
made by studanti wltti the maximum 
gain made by students of similar back- 
^ound and ability in regard to the 
same objective, 

I have suggested that there is a need 
ifor de»emphasis of the statistical con- 
cept of rekability and validity as char- 
acteristics of tests. Among professional 
measurement people there is a full real- 
' ization that a coefficient of reliability 
may mean many different things de- 
pending upon howjt is computed, It is 
also a highly variable quantity d^end- 
mg upon the group upon which it is 
eomput^. Similarly validity is the 
charaa^fistic not only of the test but of 
the situation in which the test is used 
and of the group with which it is used. 
Let me revert again to some experi- 
ences in the Cooperative Evaluation 
Study; Dealing with objectives which 
are not overtly emphasized in many 
classrooms and even in* many schools, 
it is perhaps to be expected that the 
reliability of tests would vary a great 
deal I recall one test which yielded a 
coefficient of reliability in the .6 to .7 
range in most schools but which 
suited in a coefficient of .9 when used 
in another school Just why- this was 
mie I never ascertained. In other cases 
we found that certain schools in whioh 
there .had been obvious concern witM 
critical thinking and numerous things 
in the program" directed at it, yielded 
higher rfhability coefficients for thae 
tests than other schools, in which there 
was- little, attention to these matters. In 



regard to critical tliinking, we did not 
find any really satfefactory criterion of 
critical thinking, but we did attempt to ^. 
compare the results of the tests With a 
variety of judgments of teachers and 
others, .Correspondence of teachers' 
judgments in a particular school with 
the test results bn a Critical Thinking 
Test varied extreif ely witliin a school, 
but there was even ' more variation m 
the kind of judgriients made from one^ 
school to another. As one might expect, 
in some schools where the oblective 
was regarded with indifference^ the 
very reactions of ^ teachers to tlie re- 
quest for. making judgments about 
Critical Thinking; ability almost insured > 
that the ratings -%vould be of little value. 
Rather consistently we found that 
teachers; wha had worked directly^ with 
us in tfe 'committees made judgments 
vvhich^^bfregponded closely with the 
test results^ One may^ pf ^^oiirse argue 
that this is true simply because they 
knew what was ik the t#st, but it seems 
to mtf equally plausf^le tp argue that 
as a result of working on the test they 
had a better concept of what critical 
thinking is and therefore were able to 
render more reliable and valid judg^ 
ments of its presence or absence in stu- 
dents. Clearly, reliability and validity 
' are not simply characterizations of in- 
struments or evaluatiori^^ procedures. 
* Primary concern in education must 
not be with the evaluation procedures 
but, witli the learning to which the 
evaluation procedures provide some in- 
dex. Instead of reUability of a test we 
should be concerriEd with the perma- 
nency of learntn|. Instead of the valid- 
ity of a test -wiB should be concerned 
with the utility or the relevancy of 
learning,. View<dd . in this way, these 
characteristics are related to the tests 
but they are also concepts which char- 
acterize qualities of instruction and 
predetermine mst performance, 

Finally^, I haye suggest^ that psy- 
chometrics has become conc^ned^ith 
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^eoretlgal acti^ty so iax removed from - 
the classrooni as to not only have little 
applicability but actually to alienate ^e 
classrooin teachers, I am well aware 
th&X no hard and fast line can be irawn 
between pi^e and applied rese^chi I 
should be tiie last to wish to draw one, 
On the other hand, I do beeome a little 
impatient wiA what seemi to be an 
in^easing tendency to pursue, fine 
points of testing Aeory, which go far 
beyond any utility which I can^for^ee 
for many years in actual tostructional . 
and learning situations. I should like 
to see some of our best minds in meas- 
urement confront tfie reality of educa- 
tional problems and attempt to propose = 
some hypothecs or models which are 
directly relevant to and testable in the 
classroom and in life situations. Let me 
give yOu a very simple example of what 
I have in mindp I refenedp at the begin- 
ning of this paper^ to inte^ation as one 
of the bywords of the educator, Cer- 
tainly it is one of die conce^ which 
has been given a great deal oF empha- 
sis in general education. There are 
many different concepts lOf integration, 
but I prefer to regard integration as a 
characteristic of an education person^ 
one which implies that he is a^le to in- 
terrelate everything that he knows, his 

* skills, his abilities, his beliefs and his 
values in such a way as to deal in a 
more effective way with situations in 
which he finds himself day by day. In 
short, the integrated person is a well- 
organized individual who is able to 
make the most effective use of all of 
his resources in dealing with the prob- 
lems that confront him, He is the an- , 
tithesis of our tendency in education to . 
break our objectives into subjept rhatter 
groups and even to regard critical 
thinking in science, critical difnking jfi 
social science, etc*, as distinct and ijpde- 
pendent abilities. We find ' s6irib %vi- 

, dence that the intercorrelation between 
tests or other evidence collected on 
these various abilities is rather low and 



diis reinforces our feeling that diey are 
more or less independent factors. I am 
not sure diat this must be so* May it 
not be an artifact of culture and of our 
present educational policies? 

Some time ago I proposed to myself 
and some of my colleagues the hypothet 
sis that a really good general educatioil 
program would be characterize by the 
fact that over a period of a year or two 
the intercorrelations between tests, giv- 
ing evidence of various general educa- 
tion objectives, would show a definite 
increase. In other words,/ if a person 
is really interrelating what he has 
learned there should be an increasing 
tendency to coordinate all resources on 
a particular task, It has been of interest 
in the Study to find that in one school 
where marked gains are made in the 
separate abihties tut whare the courses 
are quite distinct and where the general 
education experience is almost re- 
stricted to these courses, that die inter- 
correlations bet%veen various tests actu- 
ally decreased over a period of one or 
two years, In certain other schools, 
where there are no clear cut distinctions 
between a variety of Uving experiences 
and classroom experiences on the cam- 
pus, where instructors in qpe course are 
quite cognizant of what is going on in 
other courses and make every effort to 
interrelate them, we jBnd that over a 
similar period oi time the correlations 
among a number of different tests in- 
creased in size, There are other possi- 
bilities which might explain this and we 
have explored a number of them. At 
the present moment I have not been 
able to explain this data on any basis 
other than the original hypothesis, but 
I should still like to regard it as a, 
tentative one and - have a great deal 
jnore study of it before accepting it as 
' a simple way of ascertaining tiie extent 
of integration in an educational pro- 
gram. 

I have presented these rratiarks with 
some fear that I may be misunderstood 
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and that some people in the measiire- 
ment fleld may^ in some way or other, 
feel that 1 am brying to demean the 
importance of what has been accom- 
phihed. I assure you that this is nbt 
^e case, but it is true tiiat in working 
closely with classroom teachers over the 
period of the past few years I have 
become increasingly aware of the deep 
and increaiingly deeper suspicion with 
which many teachers regard our whole" 
field of activity. Yet in every case where 
I^have succe^fully made contac^^with 
an individual and interested him m the 
problems of evaluation and the inter- 
relationship of these to the problems of 
instructioni I have found the attitude 



to change cdmpletelyi In every case 
there con^ an awareness that Aese, 
things are not ^distinct fields of opCTa- 
tion and that evaluation has* a great ' 
deal to conttibute to tlie 'improvement 
of instruction^ — ^tliatj indeed, as I have 
already said repeatedly — evaluation 
and good instruction are indistinguish- 
able. To those of us interested in evalu- 
ation this seems to be only just recog- 
nition of the worth of our activity. 
Howeverj we need also to realize that 
thDse working in tlie field of insfruction 
can contribute to evaluation a great 
deal more thiin we have solicited from 
them. 
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Improving Evaluation of Educational Outcomes 
at tiie College Level 



PAUL DIEDERICH 



SUMMARY OF DISCUSSION 



Dr* North raised the question as to 
wheUier ins&uctors avoid re-using 
effective items for fear that students 
may learn to expect tibese items and 
make special preparation for them, Dr, 
Ebel replied that item files are kept 
and that he has seen examinations in 
which as rnany as two-thirds of the 
items had previously been used, He 
was asked whether under these circum- 
stances students anticipate these items 
and prepare for them. Dr. Ebel felt 
that this was not the case and cited the 
example of a medical college instructor 
who had exactly the same items in one 
examination for five years and had 
found no upward drift in scores. Dr. 
f GuUiksen asked whether this result was 
regarded as favorable or unfavorable 
and Dn Ebel stated that at least it 
showed that students are not as adept 



at taking advantage of die repetition of 
items as is sometimes supposed* 

Dr. Carroll asked whether there is 
any evidence v^hether the different 
categories of items listed under "rele- 
vance" actually test different things. 
He cited a factor analysis of a similar 
classification of *items at Harvard which 
indicated clusters in other ways than 
logicar analysis had supposed. Dr. Ebel 
stated that no claim was made for the 
factorial purity of the categories. He 
said that it would be impossible to 
carry out factor analyses on all his. ex- 
aminations. The analysis was essentially 
that' of contentj he saidj of what the 
examination emphasizes. Dr, Ebel re^ 
marked that a few mistakes in classi- 
fying individual items do not seem to 
affect seriously the picture of the ex- 
amination thAt tills type of analysis 
discloses. 
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Individual Versus Group Decision Making 



IRVING LORGE 



INDIVIDUAL VERSUS GROUP DECISION MAKING 



Tub FROvpRBs and maxims of any peo- 
pl@ do summarize Aeir axpariancei and 
wisdom over the genmatiom. In respect 
to individual and group behaviori of 
course, tfiere wa many such proverbs, 
Most of you have leamedi and^ at 
times, acted upon such aphorisms as 
"Two heads are better than one" and 
"In the multituda of counsallors, there 
is safetyf' By confras^ howeyars Aere 
are adages that suggest "Tnith is lost 
with too mooti debating" and "If you 
want something done, do it yourself/' 
Indeed, formal ©roup thinking4ia§ been 
# -tdisparaged by an unknown phrase- 
makers definitions of a committee as "a 
group of men that keep minmtes and 
vyasta hour^ and, informal gjoup pro- 
cedures have been reviled in Ambrose 
Bierce's definition of dkcussion as *'a 
method of confirming qthers in their 
errors" 

Though thesi succinct summary state- 
ments are in ^pontradiction, each may 
express a truetnd useful generalization. 
The apparent disagreement may be 
limited for certain kinds of tasks, or for 
specified sorts of groups, or some 
classes of conditions. Variations in the 
tasks to be done, or the organization of 
the groups to solve them, or the cir- 
cumstances under which the action is 
to be taken, may produce different out- 
comes or results. Todays social scien- 
tist, however, is not toclined to search 
the sayings of the fathers for generali- 
mtions about the differences in ©roup 
and individual performances. He tends 
to go tfifough a process of making a 



review of the UieraturB. Such a reading 
of the reseMch Uterature should, in- 
deed, reveal the range of generallza' 
tions as well as the means and evidence 
upon which they were formulated. 
When experimental studies are ap- 
proached (in the aggressive, mood) the 
variables of organization, task and 
criterion are found to be related to the 
so-cailed generali^tions about the suc- 
cesses of groups and of individuals, 

In the restricted raiige of studies Aat 
distinguish between the producti pro- 
duced by groups and by individuals, it 
soon becomes apparent that the con- 
clusions in the textbooks are broader 
than the evidence warrants. Very 
quickly comes the realization ftat in 
considering group versus individual de- 
cimon makings % group is not a group 
is not a group." In social psychology 
and in sociology, the concept of proup 
implies the sense oi two or more per- 
sons interacting among themselves to 
accomplish some objective. In fact^ the 
essential differentiation between group 
as "an assemblage or aggregate" arid 
group as "a social organism" inheres in 
the notion of interpersonal interaction., 
Groupness implies interaction. 

Yet, some of vaunted superiority of 
the group is based upon a kind of poup 
whose individual members never met 
together, and, who, as a matter of ex- 
periment, never knew that other per- 
sons were working at the same time at 
the same task. For Lnstance, Kate 
Gordon asked individual college stu- 
dents to arrange a set of weights to the 
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correct order. Each student did the task 
independently of all other students. 
From the resulting protocols, so-called 
"groups" were formed by averaging the 
rankings of fivrf (or of ten, or of fifty) 
individuals selected at random from the 
full supply. By the criterion of correla- 
tion with the actual physical order of 
the weights^ she concluded that '*the 
results of a group are distinctly superior 
to the results of the average member" 
— a conclusion which cannot come as 
a complete surprise to those conversant 
with the elementary facts of tests and^ 
measurements. 

Or again, the superiority of the group 
is founded upon a kind of group whose 
individual members meet in the same 
social setting although eacli one works 
independently upon the same task 
-without any personal interaction. A 
typical instance of such experimenta- 
tion would Seek the answer to the ques= 
tion, "Do individuals make better test 
scores working in complete isolation 
than working in a group climate?" 

More recently, the superiority of the 
group has been demonstrated on the 
basis of ad hoc groups whose individual 
members were designated to work to- 
gether to accomplish some externally 
iniposed task. For example, in the, 
widely quoted Shaw study, a college 
class was divided in half with one of 
the halves of the class arbitrarily 
formed into groups of four to work on 
problems proposed by the eKpetgmenter^ 

These three illustrations suggest thaj| 
in actuul experimental procedure^ 
groups are of different kinds. At the 
upper extreme the theorists think ;bf 
genuine interacting facevto-face 'groups 
who have a tradition of \^rking to- 
gether with the responsibility for the 
accomplislimcnt of a broad task. Solu- 
tions that such groups make have not 
been studied by psychologists. At the 
lower extreme is the concocted group 
made by some external authority who 
adds together the independent products 



of^several different isolated individuals, 
each of whorj worked alone. It is the 
solutions that such "groups'' produce 
that have been a principal source for^ 
textbook generulizatiohs, 

Moreoveri experim en tally* there is 
even wider variation in the nature of 
the tasks groups and individuals are re- 
quired to perform. Groups have been 
contrasted with individuals on tlie basis 
of sucli tasks as estimating the number 
ot beans in a bottle, as jiidging the 
aesthetic value of music, as predicting 
the date for the end of World War II, 
as selecting a bread for nutritional pur- 
poses, as plnnning a course of action 
for accepting a bequest, asJearning a 
maze, as solving a difficult mathemati- 
cal problem. 

Each diiferent kind of task has been 
subsujried under the broad term '*de- 
cision-muking," The thesaurus recog- 
nizes us related to the verb "to decide" 
the verbs, judge, conclude, ascertain, 
detcrniinc, deduce, infer, estimate, ap- 
preciate, value, assess, consider, settle, 
and choose. All strongly suggest that 
"thinking'' or "reasoning" is basic to the 
accomplishment of the task. Indeed, 
one can only wonder why the jargon- 
istic "docision^making" was substituted 
for the readily available "deciding." 

Deciding, or (not to lose face with 
my fellow workers) decision-making, 
involves suMi ideaS: as the task, its ac^ 
ceptance, a process of active search for 
, ways and means to get it done, with an 
■^ultimate selection bf a plan and conse- 
Sqtient action. The history of the study 
of thinking* indeed, is V replete with 
formulationsHike'rtiis.. As a mal/tqr pf 
fact, the early tasks or "Aufgabe" set 
by the Wurzburg School, to a degree, 
Hct the natia.'C of subsequent experi- 
mentatitm.. Basically, the Wurzburgers 
were studying what was happening in 
thinking and willing. So great was the 
attention on the process that they 
^tended to underemphasize the quaUty 
of the product or solution. For instance, 



4 6 



38 1953 INVITATIONAL CONFERENCE 



Ach asked for a free or controlled asso- 
ciation to a given word, or Buhler 
asked, "Was the theorem of Pythagoras 
known in the Middle Ages?** or "Can 
you get from Vienna to Berlin in seven 
hours?" 

Solutions for such tasks may be 
assessed readily as "right" or "wrong," 
The standard appraisal for such simple 
tLiHk problems is truth from the view- 
point of fact or actual nieasurement. 
The answers to the estimation of 
woightSi the number of peas in a bottle, 
the prediction of a specific date in the 
future, each had an objective criterion 
of correctness, In most studies of the 
procam of thinking, some rentlily avail- 
able abjective criterion for correctness 
was available. Yet most real life prob- 
lems, that must be pondered, do lack 
[in objective external criterion of cor- 
rectness or adequacy. Many problems 
of interpersonal relations are so complex 
that policy, or plan, or course of action 
cannot be evaluated as just right or 
wrong. Suuh problems range from those 
whose solutions are clearly right or 
wrong to those that can be appriysed 
only by consideration and evaluation of 
many dilFercnt consequences, 

For instance, Marjorie Shaw's pr<)b= 
Icms in the first pirt of lier experiment 
consisted of three different but closely 
related miUhematicnl puzzles; each was 
a transport problem like that involving 
the three Jealous husbands and the 
three beautiful wives wlio had to get 
across a deep river in a rowboat for 
three under the eonstraints tliat only 
husbaiids can row and tluij: the wives 
cannot be trusted in the presence of 
another man unless the husband is also 
present. Twenty=one individuals and 
five groups attempted the three prob- 
lems. In terms of just the number right, 
diere were five solutions for the indi- 
viduals over the three problems in con- 
trast with eight solutions for the^groups. 
Disregarding the possibility of differ- 
ential transfer-of-training in groups and 



in individuals^ her conclusion about 
group superibrity %vas based on com- 
paring 7.9 percent for individuals 
versus 53 percent for groups. Inci- 
dentally, no test of significance was 
made. 

In the second part of her experiment, 
the problems uivolved the rearrange- 
mcnt of the scrambled words of a final 
sentence of a prose passage, the com- 
pletion of three and a half lines of a 
sonuct, and the location of a school 
building under the constraint of mini- 
mizing jchool bus mileage. No group or 
individual solved the final two prob- 
lems, but on the sentence rearrange- 
ment four of the five groups and three 
of the seventeen individuals solved the 
problem without error. An additional 
group and seven individuals made just 
one error vvJiich involved a word re- 
versal or the placement of the word 
"there," Allliough the errors did not 
meet the criterion of perfection, they 
did not afi^cct either the smoothness or 
the sense of the sentence. Solutions^ in 
other words, could vary in their close- 
ness to the original as well as in their 
adequacy. As the problem allows for 
more and more different adequate solu- 
tions, the criterion of absolute right or 
wrong seems more and more arbitrary. 
Tlic review of the literature suggests 
that tlie differences between groups 
and individuals in decision-making may 
bo related not only to the nature of the 
task Isut also to the method for evalu- 
ating the quality of the solution. 

In general, the more complex the 
problem for decision, the niore difficult 
it is to find adequate means for the ap- 
praisal of its solutions. It was, indeed, 
fortunate that the Air Force was con- 
cerned with the appraisal of its staff 
work. For under two different con- 
tracts from the Human Resources Re- 
search Institute with cooperation of the 
Air University^ it gave the Institute of 
Psychological Research an\opportunity 
to study the differentiation between the 
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goodness of solutions of groups and of 
individuals for two very different kinds 
of problems, At one extreme, were used 
problems for which there were a known 
or kriowable soli^tion, These are desig- 
nated as "eureka" problems, At the 
other exfeemej were used problems so 
complex that it would be mpossible to 
asaertain the complete efficacy of the 
solutions. Such complex problems are 
designated as "rationa^' decision'* prob- 
lems* 

An example ol the rational decision 
type was Wilco Air Force Base for 
which a plan had to be devised to raise 
the morale of ainnen stationed at an 
isolated Air Force Base in desert coun- 
try. The base facilities wer^*inadequate 
to cara for wives and families^ of the 
airmen; the AWOL and VD rates were 
unusually high; the social and recrea- 
tfonal fapihties of the nearest town, 
some forty miles away* seemed limited 
to gamblingj drinking and prostitution. 
The elected offloials and the town's 
leading citizens were greatly disturbed, 

The, usual procedure was to admin- 
ister this problem to individuals and to 
groups, before iiistruction at the Air 
University, Th^ ^oups were ^ad hoc 
leaderless groups of ^Ix.to eight offlcerSs 
usually majors and lieutenarit'colonels, 
All solutions were to be witten in fifty 
minutes, The results for; suph;' individ- 
uals and ad hoc groupSj^jtoively naive 
in group interaction (uiaprqblem solv- 
ing techniques, demoh^tot^ t^at the 
average quality of mcftiidual de- 
cisions was superior^' quality of 
the average group d^cisionV 

The "eureka*' probjpms were adapted 
from those used by the Office' ^f Stra= 
tegic Services fai lits assBiafleht pro- 
gram. For ej^an^jp, one problem re= 
quired thi f6^l^^ll&tiofl . oF a plan of 
action fo^ ietling a cadf© o&five men 
across s^rbld mined with^pfceniitive 
en^y'phi^^- . thati c^^^ %m^^^ be 
neutffto^^npt ddlLUpJHKttefad fi^out 
was poteyitia^ ^ 



including 



* beams^ ropes, discarded truck tires, etc, 
The poups were ad hoc groups of four 
or five Air Force olficers in training, 
usually junior and senior college under- 
graduates, The solutions had to be ac- 
complished in an hour. The results 
were in sharp contrast for individuals 
and for ad hoc groups who were also 
relatively naive in group interaction 
and in problem-solving techniques* 
They showed that the average quality 
of individual plans was markedly in- , 
ferior to that of the average group* 

The "eurefia'* problems^ of course, 
could be assessed in terms of an abso- 
lute criterion of "Do , the men get 
across?" In fact, each different plan of 
action was tried out in the real field 
^ situation to estimate its work-ability on 
a pass-fail basis, In comparison, how- 
ever, the goodness of the solutions for 
the "rational decision*' on Wilco Air 
Force Base could only be estimated by 
experts judging their adequacy in terms 
of forseeable consequences. 

The "rational decision" solutions 
WTitten for the Wilco Air Force's prob- 
lem showed an unusual range, Experts 
could recognize that many solutions 
tended to treat just a specific symptom 
of the problem; on the oihm hand, ex- * 
perts appreciated the nicety .of the per- 
ceptions in some of the solutions.^ ThesCj 
indeed^ indicated not only the diagno^s 
of the many factors in the problem but 
also an anticipation of the consequences 
of the several aspects of the action plan. 
The extraordinary diversity among the 
solutions suggested that it would be 
possible to identify the factors diag- 
nosed in the plans for alleviation of the 
situation. The technique, essentially, 
requires an analysis of the decisions by 
broad areas and factors and by specific 
courses of action. 

The steps involved: first, making a 
representative sampUng from all de- 
cisions conected; second, developing a 
content analysis of each positive point 
or ide* regardless of its location in each 
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written decision; thirds arranging in a 
master scheme, the broad areas in 
which each specific point could be lo= 
cated; and fourth, assigning nunwical 
credits to each broad area and ^|hin 
it to each specific point. 

For instance, for Wilco Air Force 
problem, 300 solutions were selected at 
random. These were content-analyzed 
into broad groups such as: 

IRegulations and disciphne 

. Leadership and training 
Morale and military customs 
Problem solving procedures 
Housing and facilities 
Civilian-military relations 
Passes', leave and work scheduled 
Recreation 

Each separate point under each 
broad categoiy was identified separately 
by a code number, e.g, "communicate 
disciplinary policy to personnel" as 
point or "organize meetings and/ 
or committees of airmen to work on 
problems to formulate policy" as point 
41L 

The content analysis based on the 
reading of three hundred decisions en= 
compassed more than 95% of the 
points subsequently found in a second 
random sample of two hundred more 
decisions. The sumniaiy of the differ- 
ent points and ideas found in the five 
hundred decisions constituted the basis 
for the, subsequent quantification of the 
quality score. 

Four expert judges, after reading the 
problein, assigned numerical credits 
among the categories and to each point 
within a category under the constraint 
that the total sum for all credits woyld 
equal 100, The score for each point 
was the average of the values by the 
four independent judges. 

Each decision, then^ could be coded 
according to the content analysis 
scheme. ^The sum of the point values 
for any de"Gision could be corfeidered 
as the total quaUty score. Such a coding 



and scoring procedure, of course, al- 
lows, in addition, a frequency count of 
each specific point made by individuals . 
or by groups as well as a separate count 
of the number of broad areas in which 
such points are made, It cannot be ex- 
pected that any one individual Or 
group will make alt the points in the 
complete master key. As a matter of 
fact, the average score before instruc- 
tion is about 20 with a standard devia- 
tion of about 7. 

There is no practical way for vali- 
dating the qualify score for the solu- 
tions to .a problem as complex as Wilco 
Air Force Base. Try-out of each sug- 
gested decision is neither possible nor 
feasible. At second best, one can esti- 
mate whether the quality score does 
correlate with the Ju(|gments of inde- 
pendent experts abriut the over-all 
goodness of th^ decisions. 

To tost this* the relation between tlie 
rank-order of fifty decisions for over-all 
goodness and the quality scores was 

' estimated. Six competent judges, inde- 
perKlent of those used in establishing 
the rating system, arranged the deci- 
sions in order fromk best to worst, a 
process that took eac™ judge about four 

' -hours. The uvcrag^ inter-correlation 
among the six jud^ was ,74 so that 
the rank corjesp^ncling to the sums of 
judges' ranks \v/ve an estimated relia- 
bility of ,94, ifjje correlation between 
the rank based on the sums and the 
quality score was ,82. For other prob- 
IcmSi the results are equally satisfac- 
tory. 

In passing, it may be stated that the 
reliability of coding tlie solutions is ex- 
traordinarily high. T%vo independent 
content analysts analyzed the fifty de- 
cisions for points. The correlation be- 
tween the two analysts for the quality 
sedre was .97 without significant difler- 
^ ence between the mean quality scores 
or the standard deviations. 

The . technique of content analysis 
can be done objectivelyj reliably, and 
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quickly, The scores are iujBBciently 
sensitive to allow the measurement of 
the effects of instrjiction in problem- 
solving and in groiip dynamies, and of 
the differences between the decisions 
of individuals and groups, 

. TTie results were so illuminating that 
content analyses and scoring systems 
were also establisked for the "eureka" 
problems. The atfvantages of the con- 
tent analysis is that it allows no^ only- 
the oesignation of the kinds of points 
made by individuals and by groups but 
also serves to suggest to teachers the 
kinds of points that are often, and the 
kinds that are rarely^ proposed, In a 
seme, a content analysis of several hun- 
dred decisions is a summation of the 
best of many niinds, It tends to codify 
all the possible positive points for 
handling a problom situation. 

Against i^the panorama of such idaas, 
every speeiflc decisionj whether of an 
individual or of a group, can be ap- 
praised. Not only does the content- 
apalysis scheme allow for evaluation of 
each such decision, but also it gives the 
teacher a basis for making a diagnosis 
of faults and errors in problem solving. 
For instance* it allows the instiuctor to 
recognize that the solver placed most of 
his attention on a single factor and its 
alleviation; or again, that some solvers 
atfaek a symptom without considering 
its cause; or again, that a solver made a 
pUn without anticipating the new difTi- 
cufties that would ensue, 
- Por ex ample J the content analysis 
of the "eureka" type solutionSi when 
utilized for contrasting those of indi- 
viduals and groups, indicated that the 
individuals apparently do not check 
their ideas for getting across the road, 
for example, as well as do groups. In 
groups, each member tries to consider 
the workability of each suggestion witli 
the consequence that poor and inade= 
quate ideas are rejected. Groups, by 
comparison, produce not only? a greater 
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number but also more easily workable 
solutions for "eureka" problems, 

Groups before writing their solu- 
tions, tend to exercise a greater amount 
of criticism of the ideas suggested by 
the various members, As a groupj there- 
fore^ they reject, the apparently inade- 
quate and the clearly wrong, Individ- 
uals, hovveveri are not quite so auto- 
critical. For they do suggest fewer good 
ideas and more poor and even incon- 
sistent suggestions. 

In the 'rational decision" solutions, 
tlie behavior of groups and of indi- 
viduals ^s behavior is not very much 
different, i,e, the groups are more criti- 
cal of the ideas and suggestions of their 
fellosV'members, The result, howeverj 
with this kind of problem tends to lead 
to a reduction in the number of good 
ideas actunlly written down, This does 
not mean that good ideas were not in- 
troduced into the group deliberation. 
As a mutter of fact, in process obser- 
vation and in controlled studies, the 
group fails to specify in its report at 
least tsvo-thirds of the good ideas pro- 
duced within the group. The individual, 
however, in his less critical behavior 
does record a variety of suggestions re- 
lated to. different aspects of tlie com- 
plex problem, i,e, the individual tends 
to record ideas over the full range of' 
th^ factors in the problem. In complex 
probems with wide potential range of 
ideas and actions the tendency within 
the group toward critical review has 
the effect of constraining the area in 
whicli ideas get recorded, salvaging just 
about a tliird of the ideas contributed 
by the group. It should be stated^ how- 
ever, that these ideas, in general, are 
the more important when appraised in 
terms of the average value of the re- 
corded points, i,e„ dividing the total 
Q,P,S, by the number of dilferent points 
recorded, 

; The Quality Vnint Scure with its pre- 
reqiijsile ccnUcnt analysis gives promi^fe 
for the appraisal of high level executive 
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decisions, those by the individual ex- 
ecutive, or by staffs, or by an interac- 
tion of an executive with staffs, A com- 
plex problem really extends a good 
executive: he must appraise the situa= 
tion, discriminata and identify the es- 
sential factors, andnheft innovate plans 
and sfrategy for handling the sihiation. 
The Q.PJ, provide a basis for th^ 
appraisal of the linking of individuals 
and of groups. As such it emphasizes 
the product more than the process. 
Inference from the quality of the prod- 
uct* however, leads to useful hypothe- 
ses about group and individual prob- 
lem-solving and decision-making. 

As a method, it is different from one 
that ad^s the individual or the group to 
select Itom a limited number of alter- 
natives, a single best plan or action. 
The difference inheres in giving credit 
for the' actual number of ideas recorded. 



The dierefore, for evaluation of 

the so-called ''higher mental processes" 
is a method of greater scope and 
greater signiflcance. Of coiit^ objec- 
tive tests have been, and could be, 
produced bas^ on the selection from 
among alterhatives; and, obviously the 
results of sudh^tests will correlate quite 
highly with^ Q.F.S, appraisal Yet,^in 
aVsense, the objective tests do not esti- 
.mate adequately the ability of an in= 
dividual or of a group to produce ideas^ 
to innovate plans, and to evidence orig- 
inality in policy concept. 

Desirable as machine scoring ^ may 
be, it should not obscure the fac! that 
thg quality of high' level decision- 
making lies not only in the selection- 
from among alternatives but also in the 
capacity for devcloiDing the alternitives 
to be considered. | 
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Individual Versus #rdup Dedsion Making 



WILLIAM G. MOLLE N K 0 P F 



SUMMARY OF DISCUSSION 



^ .\ ; iDfi Langmuir raised sevCTal ques- 
^^^Mft^i one with regard to the make-up 
iff the p-oupij a seeond with regard^ to 
5>© . variabih'ly of responses Jn the two 
^ktfations, and a tiiird ^u^jip whether 
thef.results which indicated a superior- 
* ity W the individual over the .group 
1 nilgnf possibly have been an artifact 
'of the scoring system. Dr* Lorge indi- 
cated that officers had been assigned 
to grou^ systematically froni a ros^r 
that there had been a ^eater variability 
of response for individualsj with the 
best individual solution^better than the 
best group solqtioii buti)vith*the poor^ 
mt individual ^l^tiprf being distinctly 
inferior to the pb^^t ^oup solution; 
and that Ke did not ^eB^e the results 
wft'e an artifact of t^e scoring pro- 
cedure* Dr. Lorge further^ stated that 
,ttia ^oup recorder screened out some 
of the ideas of members of the ^oup» 
and kept them from getting into the 
r^rd. / V 

Dr. Andrews point)^ out that the 
recorders actions might have led to an 
apparent smaller variability of response 
for the ^oup, and wondered whether 



tape recordings might indicate to what 
extent ideas given in the ^oup situa- 
tion had riot appeared in the record 
kept by ]the grdup leai^er. Dr. Lorge 
then reported that for the several group 
sessldns which had been tape recorded, 
soflief two-thirds of the ideas observable 
in the discussions had not appeared in 
fthe groups record. In part, diis appar- 
.^nt shrinkage in number of ideas arose 
put of the group s ability to reject ob^ 
*vi6tisly wrong or distinctly poor ideas. 

Dr. Spencer referred to some work 
Q^tiQ Office^ pf^ Strategic S^vices indi- 
cating that th^ipore forkful a person,'' 
the more successful he was in 'into- 
ducing his ideas into a group discussion. 

Dr. Burk^ wondered whether the 
group ho^d^ ^en handicapped by the 
shortness ot the time, and made the 
pj^nt that rejection of poor ^3|ai' might 
be^'as important as the acfieptance of 
good ones. Dr. Lorge replied that when 
he had fried longer time limits more 
ideas were obtained both from indi- 
viduals and from ^oups, but that the 
difference between groups and Individ^ 
uals nevertheless tended to persist. 
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Problems and; procedures in Profile Analysis^ 



F . L 0 R D 



REMARKS OF THE CHAIRMAN 



I WOULD LIKE to discuss a deflrtkion of < 
terms. What do we mean by' profile , 
analysis? A profile is commonly con- > 
sidered to ba a sort of colleetiDn of 
mountains ^nd valleys arranged in a 
row, A typical problem in profile anal- 
ysis involves the comparison of two 
such mountain ranges wi t^i^ a view to 
obtaining some measurement of their 
"degree of similarity. 

Thia is not an unreasonalrti way to 
look at the problem; however, I think 
'it will , be better to approach profile" 
\ analysis from anotlier point of view. If^,^ 
■';^'e bavi the profile on 30 tests of each 
'of; 366 ihdiyiduals, thes^ profiles repre- 
^ ^Oit SpOT ^ • thir ty^ . nieasur em ents 
(each. Suqh^set'ot daW IS precisely the 
subject inaite^ df>^ bfanch;^of statistics 
called multivariate aii^ySis. I suggest, ; 
Uien, that the problems of profile anal- 
v^sis are no more nor less than the prob=^ 
l^>s of modern multivariate analysis* - 
L\t me mention very, briefly a few/^ 
of tha problems and techniques of muU 
tivaridte analydi?. Suppc^e I walk mto ; 
my dfece one morJiing arid find upon 
my desk the scores of 300 individuals 
on each of 30 tests. What procedures 
are available for dealing with these 
data? 

One important function of statistics. . 
is to redtlce a large, confusing collec- 
tion of measurements to a smaller, n^ofe 
' manageable iet of numbers, I mightN 
^'apply the principle component methofl. 
of, factor analysis and derive perhap^^ 
five, or ten measures foF each individual 



vv^hich wouy convey virtually all' the 
-ir^nnation present in the original set 
:^bfi i^thirty measui^s. A dlfferept^ ^biit 
"anypgous method of approach ihignt 
be it use an appropriate method of in-, 
v^rse factor analysis to show, that for 
certain purposes the 300 individual 
tested may be usefully grouped -into IQ 
distinctive types so that for eerjain 
purposes it is no longer necessary lb 
consider each individual sep^ately. 
, So fart I have mentionod two tech- 
niques of multivariate ^analysis which 
may^/be applied when there is no hy- 
' poth^is ' and no . further information 
available' beyomd the^Jare set.df data. 

Suppo^e/naMtr that /some of the ex- 
aminees miiy^- be ;:Jdettyfle^^^ m fi}^^-^ 
and some of them as^mechaniprfl 'Fir^^ 
' of a^^ it is likely to approprmtp fa^ 
use Hotelling's T:'Tesfc to de^fernito 
vvhether or not the t^o groups can Id© 
shovvn to be significantly Hifferent frdm 
each other on the basis t»f the test 
scores available. This statis^al test is: 
simply the multivcu'iate" analog of the 
usual test of the difference bet%veen two 
means. Hotelling's T Test may also be 
used to|determine vvhether or not it is 
plausible ip assume that a certain in- 
diviSybViy^ member of a certain group. 
A relat^'*but more complicated prob- 
■lem is' that of determining to which of 
t^vo or more groups a eertdin Individual^ 
:most plausibly belongs. This Is the maiiv 
problem discussed today by Dr. Tiede-. 
fnann. 

/. A somewhat different' problem is the 
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fdllowing; ' Al^ugh, we may be able 
to show^ViIfing 30 available scores, 
that iUO€efi£ul Amy officers c^n be dis- 
criminated ii0m unsuccessful Army 
offleer|, ;w# rtay ' be unwilling to con- 
sidef each bf 'theie 30 scores individu- 
ally^:^! would ^^}jSe some index for | 
summSrizihg , th^ infotmation contained 
in thesfe 34 |;Q^res iilsofar as it relates 
tp discriminating between a sucfcessful 
ind an, uniuocessful officer, This is one 
of the problems discussed today by Dr. 
Ahderhaltef * j 

^Pr»-Tieden>an has given us a very 
clear \ description 'of an important and 
praotieaV procedure in profile nnalysis. 
1 should Hke to mention briefly certain 
further' ■problems. Dr. Tiedeman can 
tell us /which of two or more groups a 
certain individual most resembles on 
the basis of thp data at hand. Such a 
statement* he will be the first to adniit» . 
dqes not necesiarily exactly pajrallel the 
-besiL possible recomnibndatinns for a 
vocatiQnal choice or for a Job assign- 
mentp Even from a piirely statistical 
point of view, a person'making a vocap 
tional choiae should consider not o illy 
the degree of similarity between himself 
and other members of eacH vocational 
group, but he should'also consider the 
Ipllqwing informationijf it is availabje: ^ 
• i)^ the nunjp^r bf ]c)b; opportunities «r 
openings in each 'ffeld, S) the proba= 
Isllity ■ p{^^ccess vocation, and 

3) the rewards of lu^cess in each voca- 
tion/ Given sufficient data on these con- 
siderations, the whole problem is drnqn^ 
: able tO treajijVcnt by modern statisAl^nl 
decision theory, ' ' » 

' .Without discussing recent statistital 
thebryV we are all aware of the fact 
that it wa have availableisomd rm^asurc 
of the Success pi- the degi;ea of, sUccuss^ 
of eachhof bt* jiedemaus^ mechMii^ 
and clerk typists, we mn, by niultiple 
correlatioii jniiethQds, obtain some Write- 
ment predi^iri^ the probaliic ;^uccess 
. or degree of success of a given exam- 



inee if he becomes a mechania . or a 
clerk typist. • ' / f[ 

In many situations, one would prefer 
the information given by such predic-^ 
tion to tlie. information givcn*by a cen- 
tour score; .of course^ it .would undoubt- 
edly be stiir better to have both kinds , 
of inforniatioh. Good measures of the 
success or the degree of success of large 
numbers of people working on the Job 
are in practice very difficult if not im- 
possible to obtain, so^^tfTtrf^r, yiede- 
man's procedures will doubtless be of, 
widespread practical usfi. Iti" case cri- ^ 
terion data are AvailablCj^ however, it 
should not surprise us if the conclusions , 
reiiched making use of this additional 
criterion^ infomiation are somewhat 
different than the conclusions reached 
without this information.' Additional 
relevant information is; by deflnltiorts 
always capable of yielding improved 
conclusions. ' . . . . 

i Suppose ihat in the foregoing situa- 
tion we are given still further informa^ 
tion, Suppose, in fact, that we are given 
the niimber of vacancies in each Job 
arpaj and thut we are required to select 
iridividualssto fill the speciflei^ vacan- 
cies. Here W9 have what^i.^ often call^^ 
the midtiple selectioh probleriii ynriofis 
solutions to this/ problem are available, 
tending to maximize ii^ soirie sejise the 
net success or tlie net product of the 
totality of asHignmciits. 

I siiould point out that the problems 
of profirQ : analysis are ^ not all easily^ 
solved 'currently available statistipal^ 
techhiqucs. If the data at hand dovu^t 
has^e a miiltivariate normal distribution,^ 
and cannot he triinsformed so as to have 
.such a distribution, serious difficulties 
arise. 

It is frequently asserted, epecially 
among personality psychologists^ that 
it is not enough to, work with an aver- 
age of the test scores of the examinees. 
On the contrary, one must pay attention 
to the pattern sol sc^oreSj . considering 
each score' individually in relation tO' 
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mdi of %^ o^^Si . Hie meaning of a 
•^V6n icof © on the first variable changes 
ai^drd^gVto;thi» icora observ^ on the 
s^mnd vari^IeJ This p^in% ^ ^vdew is 
cl^fy r^r^^ted, for example, by 
^in ifettace u pb n ?flie-u s e o£ multiple 
cutting scores for tiie selection of escam^ 
^ inees^ rather than the use of a com- 
posite or average score. In the field of 
profile analyiis, Ais pobit of view 
would require in the limit tfiat each 
shape or' pifttern of profile 'shoidd be 
coniidered ieparately from every other 
Aape or. ^Ite^ of profile. 

If the psyel&togist is adamant in his 
insisteiide thai each measure a pro- 
file has a different meaning depending 
on the valtje of every other measure in 
the profile, it is then clear that we are 
n%t dealing with a normal raujtivariate 
distribution* If the psychologisl can set 
up specifle hypotheses predicting the 



belia\doi' of tfie data in such a situatipnj 
these hypotheses can probably be tested 
fairly readily. IF, howeverj he must rely 
on the data to suggest hypotheses to 
him, and if the number of variables in 
Ae^ proflle ^■more'^n two QtJhr^ 
then any thoroughgo^'ng investigation 
will requir^huge numbers of cases and 
an amoiim of labor that will often sur- 
pass ^ even the potential capacities qf 
eledtronic computing equipment. 

br, ■ Anderhalt^'s rlport is relevant 
to this problem. His group made a care- 
fulj intensive, and persistent effort to 
deal with the individiial patterns of 
their profiles. With his particular data, 
hdweveii- the jnost satisfactory results 
were ipbtaine^ by the use of the linear 
discriminant ^ui^ction, The linear dis- 
criminant furfttion is, of course, notfiing 
more nor Ifss than an . optimally 
weighted average^bf the testiscores. 
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Problems and Procedures ia Profile AA^ysis 
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AN APPLICATION OF PROFILE SIMILARITY raCHNIQUES 
TO RORSCHACH DATA ON ^61 MARIN^QRPS / 
r OFFICER CANDIDATES ' V 



In TOM paper I ihall present the re- 
■•^te^ obtained when appl)dng ieye^al 
Methods of profile analyils to Roriohach 
data cbU^ted in a Maiin%Goi^i offloer 
salecM^ situation* The einphaiis in^ 
|end^(y^^e k not so much u^n the 
feul^ lind conclusions coridmiing ^tf 
^amcular ineasimng instrument ap- 
plied in this selected situation^ but 
fath^j upon fli© comparative-affective- 
ness=as rapWoaUy dCTr|onsfrated^^f 
T^alyang 'ffifi t3^e of data in terms of 
profiles of scor^ instead of as isolated 
par^ along widi a comparative evalu^ 
ation of the several mfthods of proflle 
ajialysii appli^. 

. To make the ' resiilts obtained per^ 
haps a little more inter^ting I will re^ 
\dew briefly the method of Rorpehach 
administiiation and sdsring used, as 
well as. iik setting in the Marine Corps 
jo'iemng program* l^e test is admin- 
istered through a ^oup free association 
metiiodt Cards are placed on a project 
tor in only onje position for three m^- 
utes/ Between JS to IS fiidividuals at 
a: tinie write out free r^ponses to the 
cards. FoUdwing the administrationi in- ^ 
dividual inquiries are made» these aver- 
waging about 20 minutes for each indi-^ 
vidual* The psychologists making the 
iniquiri^ tptal aO r^poni^ and do the 
inaiQ portion of all scoring* althou^ . 
die computing of ratios,' listing of 
scor^^.^p^d similar tasks are done by 



clerks under supervision, Thfe. scoring 
system used is esienti^y ditft of jEdop- 
ftr* widi B&k> .**popular&" "Somfe ^ 
mddifl^tionSj primarily in terms of tiie„^ 

t ratios usedj were made by the psy^^ 
chologista involved. In all, a total g| 
40 semrate scores are recorded for 
eadh iimividuaL ' ^ 

The candidates included in the 
screwing prograni are all s^icted from 
tiie eidiited ranks of the Marine Corps, ' 
some from combat| some from braining 
pro^amsf and. others fromfTecruit de- 
pots p Sdection for the soreening pron' 
P"am is made on the basis of ^rioiir 

.^.combinations of GCT scor^ and^coF'^^ 
lege training, plus the recommendatiojij 
of die Commanding OfiBcer. TTie lowest' 
allowable GCT is 110, ^yrfth a Allege 
de^ee, ^is group makes up a vefy 
small portion of die total involved. The 

^minimum GCT score required wi&out 

?any college fraining is 120, along with^!^ 
successful completion of a college 
equivalente test. - 

The screehing program itself nms 
diree weeks, with the major emphas^ 
being ^aced upon fielfl t>^e problms, 
^oup discussions, and a mde v^ety 
of stoess situations. Peer ratings, socio- 
mefrie ratings^ and many stand^dized 
tests are assigned to each secttbn of 
about 15 candidatei ea^ w€iik« At the ^ 
end of the screening prop'aiiij the diree 
assessorr ,who have ,observ#d ea^ 
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pohp vote^or accaptanee or rejection 
ot qandidat^i ai dom tfi© eom- 
inanding offleer, Psyelfological results 
are utiliied primufily in iituations in 
^wKie^tiie*^4 votm disapea, and then 



* and other characteristics ordinarily con- 
sidered to be asibdatad \vith a success- 
ful offlcer, g 1^. 

* ^fore attempting to analyze the 



^ ^ ranldng is ^e|^ In no 
*[. tike a^wo^^iea Rwschach protocols* 
^ Tfcose passing Ae screening pro^pm 
are r^mmended for conamissioning. 
FoUowiBg^ eoihiniisioningp all officers 
j a4lin«f approximately 20 weeks of Basic 
^ School, after which they are graded 
on b|(|i Wdersffiip Snd kcademie 
Jading. All fading done by ob- 
servers* i^; ' : 
"^irtpl^^ estimates of leadership are 
j f ^weighed heavily in tlie screening pro- 
^mHi selection. of the ^iterion groups 
# take this into .aeccftint. Dichotomous 

^'i^^®^ JB^^P^ ^^^^^ 
onP^oiy consisting of those individ- 
uals rejected ^after the screening pro- 
gram, ^nnd the qther group consisting 
. ^ ^of individuajs receiving a leadership 
'grade of^ least 90 following basic 
training. The manner in which rejec- 
tions were made in the screening course 
' and in which grading was done at the 
end ofibasic school, permitted limited 
contamination in-so-far as the Ror- 
* _ schach is concerned, So far only a 
small sample of eases^ is available upon 
which to chepl^ the suitability of the 
immediate criterion used, but the result 
is favorable, A correlation of .33 
^(wbieh. is signiflc^it) has been ob- 
: tained between overall screening re- 

suits and Korean combat ratings . 
r ' It should be pointed out that the 
, ' \Porsehach— as used in the situation — : 
j^'^ - ia^being called .upon to discriminate 
within a very select ppup. A vast ma- 
jority of The individuals involved ha^e 
GOT scores above 120, and all were 
originally recommended by their com- 
mandfAg offlcers as gooa officer po- 
tential The criterion groups are, there= 



forel the extreme|, of aA, 
ielected in terms of abili 



Rorschach data as profiles, tests of sig- 
ologiml—tttfeftaes-w^e^applie d^-to 
instance does 40 scores and ratios separately. Of the 
40 measures only one differentiated be- 
hveen tiie criterion groups, and \ this 
barely at tiie S% level of confidence, 
In view of the fact that one would ex- 
pect ^nore than twice this number of 
scores to discriminate at this level 
throu^ the operatlen of chance, it ap- 
pears safe to conclude that none of the 
40 Rorschach scores independently dif- 
ferentiates between leaders arid noh- 
leaders as defined' in this situation. 

The ; flrst ' difficulty encountered in 
analyzing the data as profiles stemmed 
from the lack of equivalence W scales, 
In a strict sense, the Ro|fphach raw 
scores did not constitute a proflla^ since 
with the raw scores, it was impossible 
to say whether an indivMual |eored ' 
"higher'' on the D scale than on the M, 
scale* etc. To overcome this difRculty 
'ull 40 scores, and ratiqp had t6 be con- 
verted to some comiron scale, In this 
process Rorschach data for 2161 indi- 
viduals were utilized. These individuals 
had all '^been ift the Marine C*ps 
screening program in the past three 
years. The diHtributions- included those 
^cepted and commissioned as well as 
those who were rejeqted for any reason. 
The ra%v scores were converted to 
normalized standards scores' with a 
mean of 100 and a standard deviation 
of 20, Since these standard scores were 
based upon ifrdividuals going through 
^ the screening program over a period of 
years, and since the standards for%d- 
' mittfng candidates to the pro-am have 
not changed, these seoi^s^ were con- 
.Mdered representative of individuals 
who might later enter the screening 
program. 

It should be ' pointed out that ms 
procedure places no restrictions upon 
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the profile of lndiyiduals=such as 
equalizing meani or vananoes. It was 
^ still poisible for one individual to 
have all low soor^, aU high icoras, or 
any ^pattering of scores. ? 

la tib© primkry piypopf of this 



paper is to illuitratf the appUoation of 
profile-similmty techniquas, it might 
be well tb refer briefly to a number of 
pth^ analyses that were made possible 
tiurou^ the ] conversion to standard 
scores^ First, /raw scor^ for each indi- 
\idual were converted to standard 
ioarmf and ^tiie m^ian score was com- 
puted for each individual in the cri- 
terion groups. Tests of signifleanees 
whep applied to die two resulting dis^ 
tributioriSp showed no significant differ- 
ence between the groups* This might 
be expected since "elevation" of profile 
has been associated with intelligencte, 
and all individuals in the two poups 
hkve GCT scores near* or abovej 120. 

Another study involved the scatter- 
ing of scores in individual profiles— or 
the flatness of profiles* For each indi- 
vidual, the range firbni th^ 5th through 
the 35th score in the profile was deter- 
mined— i.e., the 5th highest, and the 
5th lowest scorep. Distributions of these 
ranges were formed separately for the 
two criterion groups. As in the case of 
the previous study, no significant differ- 
ence was observed between the two 
groups in this respect. 

A third study involved the individual 
Rorschach scores. The 40 scores for 
each individual were ranked from 1 to 
40 on the basis of the standard scores. 
Distributions of ranks were then 
formed separately for each of the 40 
scores, and for both criterion groups. 
Tests of significance were then applied 
to determine whether any score or 
scorps differed significantly in position 
in a profile for the two groups. Signifl^ 
cance was obtained for one-fourth of all 
scores* In view of the non-significance 
of individual scores when tested using 
raw scores, this evidences the fact that 



similar raw scores for members in tihe 
criterion groups might often differenti- 
ate if considered in relation to other 
scores in the profile, 

The first attempt to analyze the data 
as a profil^ir^tended as a screening 



deviqe— utilized all 40 Rorschach meas- 
ures* Essentially it consisted of first 
sorting the 40 scores for e&ch individ= 
ual into a limited number of categories 
on the basis of the standard scores; next 
attempting to locate a sort which 
would be t^ieal of candidates in the 
high oriterioii group; then finding the 
Q-correlation between each indfi^Kual 
sort and this typical sort; and flnallyj 
noting any difference ,, which might 
occur between such Q-correlations for 
members in the two criterion groups. 
It was recognized Hiat the use of the 
Q-correlation results/in equalizing both 
means and variances in the patterns of 
all indi^idualsj which— when profiles 
being compared are flat— would tend 
to emphasize differences as well as 
any errors of measurement* However, 
the "previous analysis involving the 
range of profiles indicated that the 
scatter of individual patterns - varied 
only slightly over all individuals. Based 
upon the 40 scores in a profile, the 
variability of the flattest profile did not 
differ significantly from^ the variabihty 
of the profile with the widest, range of 
scores. The effect of equalizing means 
and variances consequently Would have 
an only negligible effect upon the out- 
comes of the analysis. 

The sort itself consisted of 9 cate- 
gories, with the highest score in the 
first category, the three next highest 
scores in the scond category, and the 
next highest 5, '7, 8, 7, 5j 3, and 1 
scores in the next 7 categories respee- 
tivelyi The use of a normal sort Svas 
used primarily to facilitate interpreta- 
tion of the resulting correlations. 

The typical sort of the high group 
was determined by computing the 
mean sort of each score for the top 
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OritOTon y groups then sorting the 
SGoras on the baste of th&ia .means. This 
ma^od did not s^w for the possibla 
occurrence of several typical sorts* how* 
.ever, tfie faet that the disfribution of 
- aadi -o£ fee 4Q_ soQrMJto the Jop cti- 



terion ©roup was unimodal gave some ^ 
support to the assumption of a unique 
sort. 

The dii^butiQns of the Q-correla- 
tions obtained from the two groups 
were inregular, however a cut-off point 
was readily obiervable at the point 
+.35, The 2 x 2 table in the upper left 
hand comer of tiie sheet previously 
passed out illustoatei the extent of the' 
discrimination obtained. If a tetraehoric 
coeffloient of correlation were com- 
puted for this table, its value would 
be +.60. 

It was noted, however, that most of 
the discrimination occurred in the mid' 
die range of scores. Below a Q-correl^- 
tion of .19 exactly the same number of 
candidates in each group could be 
found, while^or a Q-correlation above 
,50 the two groups differed by only 1 
frequency. 

When the same procedure was ap- 
plied to independent criterion groups, 
using the same typical sort, the results 
were inconsistent with the original 
analysis. The Q-correktions classified 
in the same 2x2 table as previously 
used yielded the results given in the 
upper right hand corner of the sheet 
in your hands. Here, a tetraehoric r, if 
computed, would actually yield a slight 
minus value. It was felt that the disap- 
pointing risuUs of this anafysis^ might 
be due in major part to the fact that 
many of the 40 scores used were indi- 
vidually unrehable. For further analy- 
ses it was decided to limit the number 
of variables. 

The exact basis for selection of scores 
among the 40 available, presented a 
real problem. It has already been dem- 
onsfrated that scores which do not dis- 
criminate individually might be %f 



value; Rather than use individual in- 
dices of discrimination, the basis fot 
selection was made so as to best meet 
the assumptions involved in the analy- 
ses that were to follow. The 40 scores 
were first^ screened on the basis of re- 
habihtyi WHfle no cohventlsmrt-refr— 
abihty coefficients were available for 
the Rorschach as used in this situation, 
a previous study had been completed 
wherein 100 recordi were independ- 
ently re-inqutred- Coefficients resulting 
from this analysis might be called 
"rater" or "scorer" reliabiUties. After 
retaining only those scoring categories 
with coefficients of .80 or peater, IT 
categories which were at least poten- 
tially reliable remained. I should empha- 
size the fact that scores of known un^ 
reliability were rejected, rather than 
the different procedure of accepting 
those of known high reliabiUty. 

Next, all intercorrelatioris among the 
17 remaining scores were computed. 
Through a trial and error method, 11 
scores were selected as, beihgjelatively 
independent of each other. The aver^ 
age" correlations, based upon' the ten 
eorrelations of each variable with each 
of the- other variables, ranged from 
— .045 to +.140, with 9 of the eleven 
groups being below +.100, Only 3 
individual intercorrelations making up 
' the averagds wore above .40, and only 
t of the intercorrelations were above 
.30. Eleven Rorschach scoring cate- 
gories were thus available which had 
been screened for reliabihty and which 
were relatively independent. Two 
separate analyses were then perfprmed 
using these 1 1 scores. 

In the first of these, tlie profile for 
each individual was represented as a 
' point m^ll dimensional space— follow- 
ing Cronbach' and Gleser*s geometric 
'model. Orthogonal axis were JusUfied 
in view of the low intercorrelations of 
variables. The procedure planned in- 
volved locating a point in this space 
which might be typical of high leader- 
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ihlp, and then ddtannining Uie distal 
of ^ch point for the two criterion 
groups from Uiis point/ TTie dtetance 
memme med was Crcpbach and 
Giber's D. 

cffltrpid of tfie high leadmhip 
ofiterion. poup was ielwted as the 
typical leadeiihip profile. As in the case 
of the pr^^eding analyjis, this did not 
allow for the poisibihty of, sav^al typi- 
cal leaidsrship profll^^Agato^ hpwavar, 
the unimodal disWbqtion of each of 
the 11 scores gave some aiiurance to 
^^tfae assumption of a unique typi^l pro- 
file, 

When D was obtained for all nifein-i 
bers in each ^oup^ the conventional 
t^t of iipiificance was applied. ^Ije 
non-leaders tended to be faither hbm 
this tj^ieal profile than did the kaders, 
but the difference was not itatistiqally - 
si^ificant. An examination of k rium- ' 
bar of individual profiles suggeit(sd that ■ 
a good many profiles for which the D^s 
'were almost identical^ differed coh- 
siderably in tfie patterning of sepres. 
Thus^ one might have a standard score 
of 120 on the first variable and 80 on 
the second while a second profile might 
have the scor^ in reverse. Should the 
typical coordinate be 100 for each of 
diese two variables in question, the dis- 
tance measure would be the same. 
What would differ would be the direc- 
tion of the profile from the typical pro- 
file* In the case of ordinaiy correlation 
between variables this direction would 
be an artifact of the order in which the 
individuals names were listed, but here 
a real difference might be meaningful ^ 
Consequently^ it was decided to use 
some direction indicator along with the 
^distance measure as a possible means 
fef obtaining a more discriminative 
pasure. 

Vhil^ directional cosines would have 
plvided an exacting measurei they 
w^^, ruled out as impractical for the 
number of variables involved. Instead, 



a B^m of signed valu^. were utilized . 
^e disttibution of each score was di- 
vided at plus and minus 1 probable 
deviation.. Scores above a plus 1 prob^ 
able deviation were given ft plus sign; 
thosje below a minus 1 probable 4^^a^ 
tio^wlfe given a niinus sign- and Iho^^^^ 
in between were scored zero. The di- 
rectional measure was further limited 
by the decision to use only duree di- 
mensions rather than &e ll. TOiile 
these decisions resulted in a coarser 
measure of direction dian Sectional 
eosinis, they were dictated by practical 
necessityi 

Since it was possible that a com- 
bination of variables found to disprimi- 
nate in terms %f direction might not 
discriminate as well when considered 
with a distance measure; or^^a^in, one 
which did not discriminate^©s%_ 
might discriminate when considered 
%vith distance, the two measure were 
^ considered simultarieously. For this 
purpose, the distance measure was 
categorized into one of three intervals 
on the same basis as were the individ- 
ual ^ores. In all, the use of three 
scores, each divided into 3 categories, 
and the distance measure divided into 
three poups, resulted in 81 break- 
downs. The" breakdowns were made in- 
dependently for each possible com- 
bination of 3 variables among the 11 
variables/ For each of these break- 
downs, frequencies' were noted for the 
two criterion groups. 

As a means of selecting the one 
combination of variables that best dif- 
ferentiated between the criterion 
groups, the statistic used wasi 
- D 
sum sq. of D 

where D is the difference between fre- 
quencies in comparable cells in the^ 
tables for the two criterion groups. 
While not an exact test of significance 
—and not intended as one since its ' 
purpose was to select the best among 
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Aos% available — H would result in the 
selection of Ae QOmbination for which 
tfie ratio of difference to variance of 
difference is the greatest. The statistic 
is somewhat comparable to the test of 
^l^iflmnce between correlated ^v^i- 



ableSj^e^^t that the absolute differ- 
enee is used. 

Because the data were put in 81 
categories, tiie combination selected 
had many categories with small fre- 
quencies. The method used to combine 
these categories consisted in combining 
tiie directional breakdovwis systen\ati- 
*cally until at least 8% of the sample 
fell in each remaining category. This 
process was applied only to the top 
criterion group, the grouping arrarige- ^ 
ment from the upper group >eing ap^ 
plied to data from the lower group. A 
total of 12 CLttegpries resulted, involve 
' ing the thrae signed variables and the 
. distance measure. , When Chi=square 
' was amSd^O' the' frequencies for the 
^ dtitir^ '^giTQU^^^^ beyond 
■ the .Gdl/level of .confidence was ob- 
^ tairiedj r , » 

^.v^Jibbahjliyp/^ w^f computed for 
c|i^6|oryl in eacli;case in terms of 
%ife"'prdj^biJitV.& candidate^ being 
rejectedr The^¥ ?p*^bab|litier:viahgea 
from .23 to .67 for ^bd; vai^iil cate- 
gories. ' i 

Independent criterion groups were 
selected and frequencies for the same 
12 categories tabula^d separately for 
. the two groups, "^gain probabilities 
wes4 computed foA each of the cate- 
gories. The compar*on of probabnities 
obtained are sho^ on the fpont side 
of the materials previously handed out. 
The distributions given are those of 
probabilities from original and cross 
validation groups for the 12 categories. 
The product moment correlation coeffi- 
cient computed between those pmba^ 
bilities is +.442, pointing out conrder- 
able consistency of results. 

The final analysis made involved the 
use of the hnear discriminant fuftction 



to to same 11 variables used in tiie 
preceding analysis, the raw scores on 
these variables were used radier than 
standAd scores to facilitate later use, 
. I might mention that Ae IBM equip- 
ment in New York was very useful in 



^e analysifrThe^iimultaneous-folution 
of the eleven equations involved was 
completed in about 6 minutes at a total 
cost of $35. 

The scale of measurement resulting 
when the final equation was applied to 
individual profiles ranged from approxi- 
mately -800, to +1100. The differ- 
ence between the mearis for the two 
criterion groups was signiflcant beyond 
the .001 level of confidence. 
* Scores were grouped, this time to 
terms of regular intervab of 100, and 
probabilities of rejection were obtained 
for each interval In view of the more 
regular trend noted in these probabili- 
ties, they were plotted, and a smooth 
curve fitted Jo the points. On the sec- 
ond sheet of the materials the heavy 
smooth curve is the resultant curve, 
yielding probabilities of rejection corre^ 
spending to scorep obtained from, the 
discriminant equatipn. 

A new\sample of upper and lower 
^'groups (N ^ 100 in each group) was 
again selected, and the discriminant 
equation applied to each member of 
the criterion groups. Scores were classi- 
fied in the sapie categories as ^ those 
used previously, and probabiHties.were 
determined for each category. The 
irregular hne plotted on the same area 
as the smooth curvq in the diagram on 
the second sheet presents the results of 
this cross validation. 

The distributions below the graph 
are the probabilities from the original 
analysis and the dross validatidfi. The 
first column of prdDabilities were read 
from the smoothed graph, and the sec- 
ond column of probabilities represent 
actual probabilities from the cross vali* 
dation. The'product moment coefficient 
of correlation between actual probabili- 
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tim (it. not utiU^g tbe a^^ 
curv©^ , but th© bregiSar 

. It should be borne in /^d^ 
whfle tUi eoirdatioii k h^t jdf; 
j^y^^Uo n, it prinmij^ re^ 

cc^feteney of reiults— h^f^pgB^tjj^ /iBSm tfie preening results, 



^tiOp^ ^a cdpBdent of about *ei would 
5 pfd^db i^ig^ $$me amount of improve^ 
.Jolerit?^ This flgur© is per^ 

rii^s ^ ^timate of the validity 

^ .c>( the Results than the previous conrela- 



raUier thim validity, ypJid^j^//.^c^ 
propes'Iy diould be hk t<m^ '^f f i^e^ . 
don effidenay in th& '^ ' ^ 

mmm of ^timating tii^; ;vahdl|^^^ fo^^ 
Uiis puj^se Ae moo^ifd ob^' 
taintd m the original s^ly^^ -i^fM 
to predict suc^^^ or faUurB^i^e/f^^ol^, 
*kbUity of pr^otjng spcqrii ^^^^fflluri' 
without the use of tibe Ri^b|ia€n,>.wi|' 
50^ since there wrire an/'i^ual' n 
b^ of candidate in Ba!^: 0f^^^^j 
indi^duals with a prb^abUf^j^f :^^ufe 
above 50% were pir^o^ , /ftUtir^,' 
and all indivdduali 

of failure below BQ^^^iy^^ppmlti^ii^ 
succ^es. Witti fids pir6^0uf ^/ l&li 6j, 
60,S% were correct /pi^ieyojiii fbr'^e 
mm vdidation ^oup'^ Hen^e;;^e 
provement in pre^cy@ri lp;5/^0. 
or 21%^ In a mom m^^'^M^ k^i-' 
Uon involving produot-mq^j^t^ ^]^ela^^' 



- we have 

' *Jae0i^ thaj{. |he use of a group of vari= 
^\abl#ig:' i^ohe; of which individually dis- 
■ .;sa;^inafc^^ between the criterion groupSi 
C cw- j|4ff^|intiate when consider^ as a 
' .prpflfe*. Of the profile measure us#d 
^".rthe: / Jihfear diioriminant function ap= 
/ ^eai'Sd'to be most safefactory for &e 
■!p^^ent^aituation, Thm use of Cronbach 
i jAnd-Gtoer's D when associated with 
^iomi;;, directiona measure, provided 
. sIMificant discrimination; however, it 
■y is'Jikaly ^at tiia uae of a more exacting 
: direfetional measure would have im- 
'psdyed the diicrimiiiation effected by 
".this measure* It is. likely that tiie more 
Insensitive weighting system involved in 
/ the linear discriminant fundtion—which 
p reflects direction 'as weU as distance — - 
, ij a primary determinant of iU super!- 
^ prity td ^e other methods used here. 
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D A V I D V . T I E D E M AT? 



A MODEL FOR THE PROFItl PROBLEM* 



Not fiva pftpw (2> 8, 8, 10, 

12) piibUAW^ihrtg Ai pMt Av© yws 
ar© introdueed by a mimimt diat psy- 
^olotfsti Mm fti^ wtogly b^nuM ta^ 
toasted to probto of studyin| tii© 
gimflMity of ft© psy^olo^car profile 
of M indi^ddud to iome reteenee pro;, 
fll©. This toid^endelat conaboratioa of 
toW^ to til© probta as reaiiuring to 
a p^son Wc© myself who has hem 
worldng on this problem for ft© past 
yatf. Tl^ work is betog conducted by. 
the Edu^onal^^^c^ Corporation 
imi^ * ^n^ici ;b*^en Ae Corpora- 
tion^d ffii Unltgtf Sffi^ Gov^mnent, 
rapr^ented by Dr, Lloyd G, Huffiph^ 
reyi, Director of R^earch, Personnel 
R^^ehMboratory, Human Resources^ 
R^earch ipenttfr, Laddand Air^ Force 
Base/ ftof^sor PhiUip L Rulon of 
Harvard UnivOTity is th© principal in- 
!^^tig^t^ for ftis project I^m 
. debte^'tfii Prdf^or Rulon for mmy of 
fliejdM dft^iedM this pap^, How= 
^^.^^^raibMi^f fqif^ |h^e 'rraarks 

is complete nun©.' ' 

Gronbadi andOIesCT (5), ajidKogan 

(10) todicate ftat piycholo^cal pro- 
files studied for several reasons, 
Howe^, A© hste to boA of tiiesa pa- 
pCTs have "ttie ^mpariion of an indi- 
^ddual wiA a ffoup as a common pur^ 
pose, to fte short to© available, ! 
shaU dad wife Just ftis problem, I do 
thif flot eriy bwuse a modol for a 
sat ©bsawtiona on iqdr^d^ls 
poup^ toge^CT by some .eppmion 
hmm ^Hactffistie dr cAajraotmsto 



is useful for vocation^ guidane©, ps^ 
dioto^^ ^gttosls and proffiosy, and 
anAropology, but also because my 
tt©ata©nt of data d^iffi^ to isokt© 
types whan Ae classification is «nfcTOU?fi 
ihoidd consistent vidft .tiif model 
approprit© when Aej^ifleapn is 
known, I V i ^ v*^ 

^ In ofdw to reason fronS f ^doncffete 
example^ let us pr^ume that ft© Air 
Force has administ»ed expetoentally 
m Activity Fr^Brence Imeniofy to all 
airmen inducted durifig a pven month. 
Ill© ActMty Pfef^mce Im^pry has 
a seate consisting pf ftirty p|& of ac* 
tiviUes* on© m©mber bf ft© pair betog 
an mdoor activitv and fte ofter mam- 
bar of the pair Betog an outdooTHac- 
tivity. Each ainflMi miist todicate 
preference for one activity of each pair* 
The score is the number of preferences 
for outdoor activito. The , Ze^entofy 
also Has d Second set of 35 pairs of a<h 
tivitife,,<©ne member df fte pair betog 
' a solitary activity, and fte ofter mm; 
bar of fte pair bein| a conyivial ach 
tivity. Airmen must indicate preferen^ 
for on© or iflia ofter activity . in each 
pair and the score is ft© nmnber Of 

* This resear^ was supported in ;wh©lf 
or in oart by fta United States Force 
un& cfn^L No. A* l8(600)-3ei monl- 
tdred by Director of Beiaar^* Per^nnel 
" Research Laboratory, La^and Air For^ 
Base, San itotonio, Texas, remission to 
nmtmd for rfeproduction, tanslatfon, pub- 
. taton, use and ^pps^, to whW# or In 
poit hy or for fte TJnited gtat^ Govern- 
ment . ^ ' 6 
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p%^m&^Qm for ^nvivial adti^dti^. 
Thuii Mdi ainnan hm two ictfraiy X^^ 

^ Thm nt^tm on the i4c^^{^ Tfef&rmcB 
Inmntory are not ' xpade avaUable to 
career counsdori vd^idag die aftri 
cduniding ol A@ airmenp Each airnian 
is Gounlded and aiiign^ to an Air 
Force ^ecialfy in tihe ordmary maim^. 
Aft^ a ^e tufficiant for aiim^ to ai^ 

' ittoie duti^ of their sp^ialti^ and for 
the Air Forca to Judge itB ia^f action 
with the performance of atoien in 
tii^ fpedalti^t jfttaira^^fijrW^e satis- 

* fl^. widi and were ptnOrming satis^ 
factorily in a ipecialty were dassiBed 
aocor^g to that specialty. G gfoups or 
sp^lwtifs Yeiulte^ from this eksiiflca- 
tion^fl™>Hded the bpvariate diitrib|i^ 

. tioHi^r each specialw are not all coiii-^ 
cidental, the Air Foree now has data 
for uif erring tiie regiom of an outdoor 
and ^convivial* activity preference tefCT- 
ence plane from which later satiifactpry 
and satisfied airmen in each of the^ G 
specialties arise. From inf ormatioA: suck 
as tiiis, we wish 16 determine Ae i^iH 
larity of the outdoor and convi^al ac- 
tiyity prefereii^^ of a new airrfian to 
the outdoor and convivial activity pref- 
erence of airipien',' later satisSed with, 
and satisfactorUy performing in, each 
of the G ipeoialti^ 

Now consider jBhe case of Tom Basic, 
who when tested at ^ induction indi^ 
cated preferettee for 18 outdoor and 19 
convivial activities. If we define a t^t 
plana by consbiicting Cartesian refer- 
ence axes such as tfiose in< Figure 1, we 
may indicate all tiie *trifonnation con- 
cerning the outdoor and convivial ac- 
tivity preferences f f Tom Basic by 
placing a point iti Ae test plafia at tiia 
Intersection of a line parallel , to the 
convivial axis toough the point repre- 
senting 18 outdoor activity preferences 
and a line parallel to the outdoor axis 
tiirou^ die point representing 19 con- 
vivial activity preferences^ The informa- 
tion that Tom Basic'prefers 18 outdoor 



and 19 convivial activitias is t^en called 
coordimtBs of the point for Tom 
Basic, Thus, the scor^ X| where i tak^ 
the values 1 and 2 define a point in die 
test, plane i Gronbach has referred to 

1 m odal jti -afr Iftagt three his pa« 

pers (3, 4 S). Cattell (1) refers to die 
mddel in his bopk on personality* 
Osgood and Suci (12) found the model 
usef^ in r solving some of die prob- 
lems concemed in the an^yiis of 
semantic data, and Gaier an4 t^f (8) 
nnention the model in a recent^-ftf\dew, 
1 This form of r^reientetion ©f ;^© set 
of scores is familiar to psyehologistSj 
since they have been making itatter 
dia^^ms for some timg. However, it 
seems to be only receifjl^^^diat consist- 
^v ent attention has ^beefl if given to diis 
conceptualization of the profile of an 
aiiman. 

We may, of course, indicate the two 
cpordinates for Tom Basic on paraMel 
, refarence axes as in Figure 2. If we do 
; thiSj we have a p^j^fi 
^ ^Let us^uppoie-^ among the air- 
"^I^W^^.td %tehl^t^^- Force adminis- 
fered the, Ap*|b^y^^^ 
experiihentally,!; 8^ \of them lat^ be- 
came ^tisfaetpty and satisfied Glerk- 
Typists. The" records, of outdoor andi 
convivial activity preferences pf these 
85 ainnen are sorted from the^r^bordsj 
for all the airmen tested in die; experi- 
ment and recorded in a; roster ^sutm , as 
that of Table L If the outdobr and con- 
vivial activity pre£erenitf0sjbf{d^^e later 
satisfactory and sati|B4^^1erfcTypisti 
are taken from Table 1 and ^ ^closed in 
two sets of parallel lin^ as they are in 
Figure 3| and if die labels for rows and 
/cOlurtns nre deletedj the result is caljed 
a rnaiHXi The score matrix which has 
beftV de^ghated Xj in Figure 3 conveys 
eKactly the same information as the 
roster for Clerk-Typists reported in 
Table 1* The mattixj howeverj has an 
advantage over the roster in Table 1; 
it immediately implies that the pairs of 
numbers like (10, 22), (14, 17), and 
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§0 on,^@ ^rdinatti of 85 pdflb in 
ttie plm©.^ Cart^ian repr^entation 
of tibe maME Is given in Figure 4* 

Fi^e 4 t^tmm^ aU information 
avaflable to Ae Afr Force conceming 
tto m^^ fag ^ egp r^ion of prefy epce 



for outdoor and convivid activity tor 
late prf^enG© in, iatisf action with, and 
sa^aetbry ^fonnance in, th© Ctefc- 
Typist ip^Jty* The figure suggeftsi 
(1) ^|t latof satiilaatory and satiified 
Cl^k-Typiitt tend to be found in the 
up^CT Irft portion of Ae test plane; (2) 
jotot pretereno^ for. otitdoor 
and convivial activitiei of later satis- 
factory and satisfied Clerk=Typists are 
more dens© in ttie region of tfie t€st 
plane riw tfie CBntroid of the Clerk- 
Typist ^oup; (3) that the demity of 
outd<^r and c»nvivial activity prefer- 
ence of Gtek-Typists b^m^ less as 
one mov^ fai 'all direcHons from the 
centroid; and (4) that dispersion along 
the new reference axis labolled^iig 
more widespread; Siari'dispersio^ along 

the new referfencf Wte ^ly?* 

We noted earJfa.that a profile re^ 
suited from representation of the co- 
ordinates of thefOutdoQf and convivial 
activity prefereriees of Tom Basic on 
pai^llal ref^enee ^es. Ift a similar way 
we may form Ae profiles of outdoor and 
convivial ac^vity preferences f6r each 
of the as Clerk-Typists as in Figure 6. 
However, in Figme 5 we have not con- 
nected ^e outdoor activity coordinate 
d£ each airman with his CQoyivial ac- 
tivity coordinate. Therefore* it is im- 
possible to tell in Figure ^ which out- 
door activity preference are associated 
with which convivial activity prefer- 
pnces. Hence, we have Frofile Froblenv 
i- How may the joint preferences of 
jea^ amnan be representedt To repre- 
ient flie Joint preference of feach airman 
on a proffle results in such a mess be- 
tween the two profile stalks that tiie 
profile for Tom Basic when pyotted with 
r^ect to the profile for Clerk-Typists 
b^me obicure, ^ 



B^ause of this fact, we fr^uently 
indicate only two pifofll^^hen we wish ^ 
to compare the profiW 0r Tom^ Ba^^^ 
with tiiose for Clerk^ypfetsi vlfle two 
profiles are the profile for Tom Basic 
and the profile for the centroid of tiie 
Clerk-Typist ^oup^^dicated-in-Fig^™ 
ure e. Figure 6 suggests immediately 
Frofih Froblem 2i How may the simi- 
larity of in airman to airmen in a par- 
ticular specialty be e^ressed? In order 
to answer this miestiwi let us note first 
that all of the 85 airmen vi^hose outdoor 
and 9Dnvi%l agtivity preferences are 
representecnn me test plane in Ffgure 
4 and on profile axe in Figure 5 have 
the same label Each onrf^.of these 85 
airmen is a later st^jtisfactory arid satis- 
/fled Clerk-Typist, And yet/ Figures 4 
and 3 indicate that these 85 airmen ex- 
pressed outdoor and convivial activity 
preferences at the time of induction 
that were neither coincidental nor col- 
linear. Thfe condition has occurred so 
frequently in experience and^I^m 
sure in yours as yrfell that I sugg^t we 
accept itj su^maifte'e it, arid even give% " 
it the fOTuar^ame^^^ Lets have 

tjie akidi^'rccitt as foHbivs i 
AxioM /;--^oints repr^enting pairs 
"^o£ piycholo^cal observations on air- , 
men gfoypedvaccording to a common 
desighatioh^ will ordinarily' be dis- 
persed about the centroid /in a space 
of dimensionality two. 
This axiom does no more than formalize 
for bivariate data the principles of in- 
dividual differences and reliability that 
most of us accept. 

Application of this axiom to the ex- 
ample under . discussion refyl^ in a 
feeling of relaxation conceming the 
bivariate dispersion of outdoorfand con- 
vivial activity preferences of me 85 air- 
men later classified as satBifactory and 
satisfied Clerk-Typists as repr^ent^ in 
Figure 4. All of ^^e airmen are of tiie 
same kind. However, Ae outdoor and 
convivial activity pref^ences of sU of 
these airmen are not of the same kind? 
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Oitf task may then be ttatod ^is ymyi 
Mom yk<^ is it nmdom ppint 

(Xjt ^) wbm eoQl^dfaAMfm the 

kM abodat€d with itP , 
r An a?^^g*» *h q"*^ 

Qusly^.eur d^Udq^ of tfaoo^ri^ can^ 
not he wo it^onli l^at pbftte must Jbe 
« c^ocidentiL U mdkM AAMon of siml- 
<Udrity w&e emplc^4d it wD^d r^ult to 
.mining dhAM ^JCT^tta late sa^a^ 
tey and ^^sd- darfc'^^pisti, &on- 
badi md Qlmiiffi taim of ee^MMy 
(B, $y md CM^'s .oc^^mt of 
pattern gimilarify (i) fa^t ss stoUar 
. aU pokite lybg on a drda otnt^^ at 
die ^nMid of i^e ^Oyp, 1 have 
r^dy indi^t^^^t Mdd# diipmion 
dong df@ Sii^ a]^ tii. Figitfe 4 ^an 
along tfie %|32 a^di is to be #l^ected In 
^e outdoor ai^ '^nvivial aotivity pref- 
^.frenc^ of later ^tisf actoty iatiafiad' 
Gtek-Typisfei Cpns^uentl^. if I war© 
to define B^ ri^n^lsix o& a 

, cdrala, oenter^ at the o^b-oid of 
fciWk-iTyptot specteltyi J would be 
treating Us similar ^in!s that occur 
wUh jdiff^ent relative fr^uenciei /in 
&e profile lor Jat^ satisfactory and 
satiifiid OtefcT)^iits, Ii^ order to over= 
com# diis difficulty, I dpfina iimilarity 
in tdrriis of ^ual probability of oocur- 
renoa of a point within a specialty, ^is 
definition is a second fi^damantal tenet 
of my rasQfark#iand cxD^^uently I also 
state jt atan axiom^ 

Axiofn II*— Similar' profiles within a 
pecialty ar# those that occur ^th 
be same probability. 
|n ord^ to apply Uiii axiom to a 
coinipanipn of Ae oi|tdoor and convivial 
activity praf^enc^,.of Tom Basid with 
those of lat^ satisfactory and satisfied 
Clerk-Typists, it Is nec^saty tb specify 
the regions of ^ual probability in the 
t^ plane of Figure 4* Obviouslyi it is 
impos^fble to dtf tfiis from tiie empirical 
data of f^igure 4 itieUp since Uiare is no 
consistent location of polnte that occur 



spf 



with a ^^uaA^^ 3, a^aqu^^ of 2, 
or a^^^uen^ of li Th^efor^ ^e turn 
to a ^eo^ti^ -distribution. 

Figure S indi^^ tbat ifc^ ii r^on^ 
■ abie to BSiimie tibat both thQ outdoor 



lurgQ '^mple < $f ,is,tm sa^act^ and 
satisfiad Ctek-I^l^ are dMributod^ 
nomiaUy» Fi^ug«4 i^di^t^ ^t tibe^ 
re^^^on l^^f^o^^tdy t Hhw. 



"Whm bodi' 
bivariate 
a bivudata ' ^no) 




m partainp 
;^!^Mbutod in 
]pann^« Consa- 

ij^ef&fBm^s of a 
, ^ sMrf&ct^ and 

uted nornMy^ m^nj .of t)M of 
this ^mple and mth dispm'Mn matfb£\ 
of tt^ of ihiB mmplB, , ' . * 
. Th^ dispfriign mati^ may; be comii 
puted directly from &^ mw., score 
matrix Xi given as Figure 3 by^dg^^g 
bolb a mabnx of m^ns as illi^^spd in 
Equation 1 and a ma&u>of n^rai^r 'of 
cas^ all illustrated in Equation 4, and 
performing^th© matrix Operations indi= 
catad by Equations 2, 3, and S* For dia 
Clerk-Typist data^ the operations result 
in tihe topenion matrix ^ven as Equa- 
tion 6. n^e mahix indicitei tiiat di^^- 
sion along the outdoor activity prafar- 
ence reference axis is somewhat ^eater 
^an disperiion along the convivial ac^ 
tivity : preference reference axis, and 
that outdwr and convivial acti^fy prrf" 
erancaa are related positively to a small 
dd^ee. Consequently, variation along 
the referenca axis iCjjg in Figure 4 is 
pisn variation* along die refer^ 
enca axis Xi^^ in diat ^ura, ^a diipar- 
sion matrix is, of course, a direct func^ 
tion of Ae raw scofe matrix since it was 
computed ' from Equation , 7, the ex- 
panded way of writing Equation 5* 
Tliui, when freated, in tiie right way, 
tlfa score inabrix X| reports aU the in- 
formation concerning outdoor and con- 
vivial preferences ttiat is of psydiolo^- 
cal significance for inferring later duty 
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* as a CtetTypiit I lay ftJi vjdft ^itt- 
plate co^denee Ibecaui© to indoor- 
outdoor and soUtary-cotfvdvtel scales of 
the AatMty Prefm'Bnce tnpmiory a 
fluent of Professor Rulon's imagina- 
tlfi^ fls Inflneneed by Prof assor Kdl^'i 
work on setivity preference and data 
for Clerk-Typtots were obtataed by ; 
throwtag die© to approximate sp^cifi^- 
tion of . Ae matrix oif meani and tiiB dm- 
persion rtiatrix* ' ' 
~ In a bivariate normal disttibutiop Ui® 
probabiU^, P(Xi, Xg). that a^p0^, 
(Xi, Xs)i drawn at random from the 
bivariate disttibution will be from a 
( smaU area surroimding tiie point is 
given by Equation 8, where |he symbols 
a, and p h^ve tfieir usual meaning 
of population mean, standard devia- 
tion, and correlation respectively. For 
fixed values of tfiese parameters the 
probability dependi dnly upon die 
quantity in die wiggly bracked of 
1, Equation 8. This quantity is copied in 
' Equation 9 and is balled x!. following 
Pearson (13). / - 

^ Since wa do not knbw the values oi: 
the pararrveters in Equation 9,^ we 
choose the^ maximum likelihood esti- 
mates, sample meahs, standard devia- 
tions, and correlatidn. Computation of 
the correlation coeffleient itself is not 
necessary, because if the mate vari- 
able xi is defined as it is in Equation 
10, may be con^piited directly from 
Equation 11. I have:-v;nritten tlie sub- 
script 1 after ^ Equation 11 because 
this is the X" the deviation scores 
with respect to die means of the Clerk- 
^■^ist, or first, specialty. Equation IL 
^Mieates that diere are a number of 
vffiies of die mattix variable that 
yield die same value of x^. The locuS 
of points wi A equgl values of X" 
that of an ellipse with center at^ the 
centrofd of die Clerk-Typist group. The 
ellipsa is symmebric about the Kelley 
(9) principle components of the Clerk- 
Typist data. Two of the set of iso-fre- 
quency ellipses are shawn in Figure 7, 



I have Uiid the symbol *® denote 
die value^ obtained from the triplf 
mabte pr^uct in Equation 11 because 
the distribution of diis x^ is die same as 
the disttibution of "^^d ordinarily in 
tff ts of independence or jn^ ftsts of 



goodn^l of fit: The. number iblf depeei y 
of freedom ^ssofiiated with tMi^X^ is the 
number of variates, in our illustation, . 
two. Consequently, a value of x^ 
pute^ ftpm Equation 11 may be evalu- 
ated' in terms of probabiyty from any 
of the tabte of x^ commonly available. ^ 
If ^eater accuracy for the relative fre- 
quency of the x-*s which exceed a given . 

is desLred, use Table IX of Pearson's 
fables for Stamiaiam and BiomeM- 
cians. Part pp. 22^23). 

Thus, Vx^ tl^^ distfflice in the 
Mahalanobis (11) sense of a point from 
die dentroid for a specialty whjch is 
consistent with Axioms I and 11* and 
contains all of the psychological mean^ 
ing for later presence in, satisfaction 
with, and satiifaqtory performance ip, 
the Clerk-Typist specialty inherent in 
the matrix Xt, If x' commuted for 
every point in the bivariate Gaussian 
distribution, and if a frequency distrip 
bution of these x^*^ ^ determined, it if 
possible to compute die percentile rank 
of each value, If computation is done 
such that a x~ coffesponds to ^ 

the percentile rank jlOO, the percentile 
rank will indicate the per cent of points^ 
in the bivariate distribution bfyond^e 
contour ellipse upon which a p>int with 
a given x" value falls. For this reason* 
Dr. Rulon, Dr. Bryan and I (15) have 
called this percentile rank a centour 
score. The centour score is not only a 
percentile index of distance but also an 
estimate of the relative frequency with 
.which the outdoor and cdnviviaL ac- 
tivity preferences of Clerk-Typists will 
exceed th$ P^*"^ representing 

the outdoor and convivial activity pref- 
erences of a given airman, provided 
outdoor and convivial activity prefer- 
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;b# ^i^ed to ^m@' sp^^^ 



1^ madt ^vdi3^ of Ctefc^tot X* 
^ eQuU to^' gMtv^u tibe vidue afiO- 

{ Jtf wi riowv t^ for 
[■ tt© 18 outdoor aafl 19 i^n^i^ a^^ 
/ prrfeitoc^ of Ttwa Bapio, w© get 

/ pasrkon'of th© pjfOT© Baiic 
^^th those for kt^ sa^faato]^ and 
^§«^€d CfiH-W-T^iite,^ He x-*s for ap-^ 



Btewtf 8. lie matrb Xp' "ta \y^tten 
^^bout ioflueaee on ma^x Xj. Ibe 
ngotm indi^tad in tbe lows b{ 
matriJE in Figure 8 repr^ent ^e' cb- 
ordinate of potot in a test plabe such 
m Figurg 9 for of thg later gatii' 
TacBty^KTiaffiBWTSrs^T^I^toWSr; 
^ Figi^e 6 is conataetfd by addtog tad. 
pQinte to tbe prevdqos Figure ^4. Figurt' 
9 indieates &at m§ location of the 
po&t repreitoting the oufdoor and eo^ 
Sdvial aettvity pr^f^ened of an afctoiffl 
' ih tfie^ teat plane to related to lat^ 
identjflG^ie^^^ a ^Clerk-Tj^iit or an 
A & EMtehanie^FifWreJ indScat^ ^o 
ttiat tfie meaning of outdoor and .don^ 
vivid . aetivity preferences is different 



prdrisnaMy 15 p^ Arit of tibe Wwfc ^syohotogically rnicTng later satofactoo^ 




saHsiac^ry vand satisfled. dlak-Typists 
es^d tiUi^^Wue of x^ Tkm, Tom 
Baaiq impressed pr^^enoA for outdoor 
^and 09nvi\1al aGtt\^ti^ of a? type whose 
disdrepanqy from tihe crafroid of &e 
Clerk*T^ilt specialty was exceeded by 
^ dis^^ri^d^ of 15 per 0|pt of the 
later iaysfactb]^ and satis^i C|erk- 
■rTypbjts; If - all ^miea^ wiA ; a d^^^ 
^an^ thfa large or lafg^ were extfuded 
l^bin tib© derk-Typiit ^ecialty, ap- 
prp34fflately 15 p^ cent of thos$E' Whd 
wot^j^ nonnally ^ beqome satisf a^tbry 
and salSsfled plerk-T^^iste later i^ouM 
be WEoludW from tiiat speciaity* ; 

The eentotu' score permits Interpreta- 
dons of this nature for b1\ points in the 
plue^ Cenfour scores may be sum- 
zed in a teble such as. Table S.' 
Table 2 ^ntaiiis the p^eentil© tquiva- 
lenli ot the distance of a sample of 
pbiato in ^e t^t plane from the cen- 
trbicL for the Ctok-Typiat specialty. 
§iieh^informatfon is quite useful in die 
^feei^ cbimseling of airaien* 

This mbdd is devdoped in terms of 
a! sin^ specialty, '^^en Ac outdoor 
and convivial ' acti^dty preferences of 
airmen who later bebamb saUsfaetory 
and satisfied Air^aft & fingine ; Me- 
chanics are asseitobled/ they .pay be 
Ai^tten as a new matrix of sepces as in 



and jatisflfd A ^ 1 Machanics than it 
was ampi^ later Satisfaetoryj,aiid satis= 
fled Clerk-Typists. Among kter satis- 
fied and satisfactory A & E Mech^cs 
dispersion of outdoor arfd convivial ae^ 
tivity preferences is greater along Ae 
a^di^Xg)^^ than it i| along the axfe Xgaa. 
This infonnation is implied in die ma- 
brix Xgj. but is not explicit 

In order to make the inf ormatioh of 
Xa expUeiti it is necessary to form bodi 
a maWx of means arid a matrix of num- 
ber of A & E MechaAics as has been 
dbnq in Equations 1^ arid 14 respec; 
tiveiy* The diipjersion matrix for- tiie 
A & E Mechanic group/ may then be 
computed from Equations 13 and 15* 
If the roafrix variable Xg is fonned for 
every pair oh scores and acbord- 
ii\g to Equation 16^ equal fr^uency 
points , in the test space for the A ^ E 
Mechanic specialty he on one of the set 
of homothetic ellipies given by Equa- 
tion 17, provided the bivariate distri- 
bution of outdoor and convivial activity 
preferences of A & E Meghan^ is rea- 
sonably nonnaL Addition of two of this 
new set of homothetic efflpses to Figure 
7 results in iFigure I'K 

For Tom 'Basicp 0faluatioh of Equa- 
tipn 17'^vfes,OJ629 as the yalue of y?^ 
for the 'comparison of h^ poirifjWim 
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those for A & E MechahiQi, This 
exceeded by tha x^'^ approximately 
88 per cent of the A & E Mechanic,; 
88 is the. cento^ score for Tom Basic s.;; , 
point compa^ with the A & E Me- 
chanic distribution* It indicates that th_f 
X^ of Tom Basio'i point is exceeded 
quito frequently by the x^'^ of the 
pointji of later satisfactory and satis- 
fied A E Mechanics. 

W e now know that the centour score 
of the point for Tom Basic is 15, when 
compared to the later satisfactory and 
satisjBed Clerk-Typists; ^nd 88, when 
.compared with the later satisfactory 
and satisfied A & E Mechanics. Torn, s 
preferences for outdoor and convivial 
activities arc more typical of those of 
later satisfactory and satisfied A & E 
Mechanics than they are^ of later satis= 
factory and satisfied Clerk-Typists. 

Centour scores for comparison with 
A & E Mechanics ' may be added to 
Table 2 as they have been in Table 3, 
Table 3 contains all information from 
the outdoot and. convivial activity pref= 
erences expressed by an ainnan at in- 
duction for inferring later presence in 
either the Clerk-Typist or the A & 
Mechanic specialty. 

In this brief consideration of a sec- 
ond specialty I have itjdicated sufflcient 
steps for extension of 'the logic to 3, 4, 
and so on to the last or G group by 
means of induction. Computations 
specified by Equations 12-17 are simply 
repeated for each new group added, 
__Tabl0-^-may be~~angniented by: a new 
row in eaclpof the row blfJcks associ- 
ated with a particular convivial activity 
preference score for each group. 

The centour score model for infci ring 
group membership provides implicit 
answers for Profile Problems 3 and 4. 
Profile Problem 3 is: On what type of 
scales should profiles be represented? 
Centour scores are based upon the ns= 
, sumption that tlie bivariate distribution 
of each group is normal, Thus, the 
scale on which profiles arc represented 



and the scale on which Cartesian rep- 
resentation of profile inf ormation is 
plotted should be such that an approxi- 
mate bivadate normal distribution re- 
sults in each group. When the distribu-*, 
tion of each variate is normal in the 
groupj and when regression is linear in 
the grotfp, a bivariate nomial disfribu- 
tion results for the group. Therefore, 
the basic requirements for the scale on 
which each \ ariate is represented are 
that it produce an approximately nor- 
mal distribution in each of the groups 
with which the profile for an airman is 
to be compared, and that it be related 
linearly to other variates in each group. 
Raw scores themselves may fulfill these - 
properties. However,^ if the raw scores . 
do 'not fuflll these properties, some 
transformation similar to the transfor-^- 
mation used by Flanagan (7) in the 
construction of his Sealed Scores for 
the Cooperative Tests may provide the 
> desired conditions. The transformation 
is necessary for only those variates that 
do not fulfill both conditions J^f^ach 
group, 

rrofile Problem 4 is: On which 
group should profiles be standardised? 
In the comparison of, a point with spe- 
cialty 1 the CMDmparison should be in 
terms of scores standardized on spe- 
cialty L In comparison of a point with 
scores for specialty 2, the standardiza= 
tion should be in terms of scores for 
specialty 2» and so oui to the last, or 
G^^S specialty. 

The centour score model has a fur- 
ther advantage of considerable, impor- 
tance. Equation 17 does not depend on 
^ the number of variates on which each 
individual is observed. Equations 12-16 
need only be augmented appropriately 
for every variate added. Pearson (13, 
p, xxiv) has indicated that all points on 
the elhpsoid defined by the generalized 
foHm of Equation 17 are equally prob- 
able in a multivariate normal distribu- 
tion. The vahic of the centour score can 
then be determined from a table of 
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with de^e^ of freedom equal to the die specialty b^re classffication is 

number of variates. ^ ^from Peaison's done. If ^e ^nfouf stores are to be 

(13) tables. . „v used for guidance purposes^ it is my 

The volumes necessary to Bind tfie feeling that each counselor should ex- 

• - tables of centour scores that would re- ercise his own Jud^ent in dealing with 

suit from even as few as four or five the question of the ratio of men in each 

variates and three or four groups will speciftlty* ' - * 

* unddtibtedly lead 0 immediati eflorts In the development of the model I 

to reduce tiie number of variates. In hdVe stkted dvo axioips. One a5pom ex- 

my opinibni these efforts "will be most =^ pressed the f aitfi that the observations 

fruitful if they starf 'from this model of individaals'shaHng a common elassi- 

if* and evaluate the efiBdency of th^ re- flcation-will eontaip multivariate diiprf' 

duced nujpgiber of variates with this s^n of the observations ^pbul the een- 

model^lis the standard of efficiency. The t&id. The second axiqm defined as 

Rao-Tukey-Bryan multiple discriminant similar those po^|i within the con^on 

function shomd Aot ^e overlooked in classification that joccur^with the same 

efforts atjedtfct ion. ^ * probability. With th^e two tenets in 

The cBntour score conversion of x" mindi I suggest that you trace the locul ^ ^ 

is, I believe, the index most useful in of points in asfest plane rejultingffrom 

guidance work^ especially of a voca- efforts to de®ibe psychological types 

tional guidrfhce nature. In work of this by person to pers^ comparison, mean 

kind, I believe that the counselor ^catterj vocabulaiif^ scatter, the selection 

should be aware of thi realities of the of similar pEpflles, Crontoch and 

ratios of individuals in various jobs and GleserV^'(5) index of^distaficej Catteirs 

i that th^e realiti^ should ^fluence the ^^2) coefficient of pattern similarity, and 

■ interpgfpttftion of centour scores. How- ^uMas" (6) coefficient of profile simi- 

ever, I do not believe that th6 ratios larity. If you ^o this^ you.udll nnd the 
given by numbers currently in jobs, loci 'if points inconsistent with either* 

should have the influence on the inter= 4- or both of the axioms I have stated* In. 

pretatipn of centour scores that they this situation, youtipre forced to abandon 

would/ have if the ratios were , to be either the techniques a^^ a n^fflitis- of 

incoi^orated in this model and they isolating types or the axioms as a de-"'v 
were to affect classification regioifs in a^ scription of \vfiat you would expect th^; , 

test space such as^ that depicted in Fig= bivariate distribuyons for a type to be. 

ore 10. The intrchduction of tlicse ratios Personally, I ImvKquestioned the tech- ' 

into the interpretation of centour scores niques, ' 

is, of coursCj necessary for the classifl^ I ruj icated earlier that I considered 

cation or assignment of men instead of it desirable to have an adequ^e model 

their guidance. The introduction of for inferring a known classification from 

these ratios will increase^ the efficiency a knovy ledge of 'psychological charac- 

of classification of men. However^ teristic^ beqause I felt that efforts to 

boundaries of these classification re- develop an unknown classification sys- 

gions are still fuitotions of Equation 17. tern on the basis of Multivariate obser- . 
Therefore^ thte equation is one fha^ ' vations should be consistent with the 

must be determined whether the data other niodck --. ; 

are to be used for guidance purposes I wish that I could offer a solution 

or for classification purposes. If the data to the problem of resolving a^multivari- 

are to be used for classification pur- ate set of N points into the mulHvariate 

poses, the centour scores need to be distributions of G types which is con- 
modified by the proportion of men ig- sistent with my two axioms. Someday 

* 
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i:^ope dfet someone wUl. Iftoyorie is 
intereitri in blinking a^oul ^ pcab- 
^lem, I can at least share a lead with 
him. On pkges 300^06 of Rao's recent 
book (14) yoi^viU find tfiat Rao treats 
die probem of resol^^ a mixed series 
into two Gaussian cJ^pnents. Rao in- 
flyicat^thlt a solution of this problem 
" for t^ease of a single variie and two 
groAs in terms of the^method of mo- 
meS v^as discussed b^Karl Pftrson 
as earVl^ 1894. The gblutiofl^of this 
problem requires estimatioi^of the two 
joeans, the t]^^tandard deviaUon_s, 



ai^ the proportion of mixture froilft die 
data -on the single^ yariate of the mixed 
series. Rao gives an adaptation of Pear- 
son s method. Rao further discusses the 
problem of sexing osteometric materidl 
on the basis of multiple measurements. 
These solutions should, I thLnk, receive 
tlije attention of more psychblogists. 
Before, closing, I acknowledge Dr. 

^Lord's kindness in dirfecting my atten- 
tion to an error in my original state- 

. ment of the sufficient ^test for>bivar»te 
nortnahty and in'germitting me to cor= 
rect the error before reading my paper. 
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FIGURES, TABLES, EQUATIONS, AND REFERENCES FOR 
A MODEL FOR TOE PROFILE PROBLEM 

Figure 1 

Cartesian Representation: 
Outddbr and Conviviar^Activity Preferences — ^ 
Tom Basic 



35 T 



30 - 



£5 " 




19 -- 



5 



a 



Number of Activitij Freferences 



• (Tom Basic) 



10 ,,, 15 
Outdoor 



20 



25 



30 



72" 
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Figure 2 



Profile Representation! 
Outdoor and Convivial Activity Preferences— 
. Tom Basic 



Number of Activity Preferences 
Outdoor ; Convivial 

35 T 



30 T 



25-- 



20-- 



30 



25" 



20-H,v 

(Tom, Basic) 



15" 



15-- 



10-- 



10-- 



5-- 



5-- 



O-L 



,0 J. 
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Table 1 

Outdoor and Convivial Activity Preferences of 
Later Satisfactory and Satisfied Clerk-Typists 



Airman No. 
1 
2 

-3 



No. of Activity Preferences 
Outdoor Cormvial 



10 
14 

19 



22 
17 

a3 



85 



16 



24 



Figure 3 



Matrix Representation: 
Outdoor and Convivial Activity Preferences— 
Clerk-Typists 



10 
14 
19 



22 
17 

33 



(85 rowa) 



16 



24 
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^ Figure 4 

eartesian Representation i 
OutdTOrtodeo^ivJaiA^ity Pref^^^ , 
. ■ V • ^ Clerk-Typists] ' . ' / . 

^, . -t^umber of Activinj tref^^^ . ' 
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Figure 5 

. jroflle Representation: 

Outdoor and Con\ivial Activity Preferences— 
Clerk-Typists . / 

f Numb&f of Activity FreferBnces 
Outdoor Convivial 



"25 



• •i-20 



"... • 

•^S"*. -t'*^ S m 

m m 4 » • « * > 

• ,4 # • • 



10 




f 

ft) 
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Figure 6 

Proflle Representation: 
Outdoor and Convivial Activity Preferences- 
Tom Basic and Clerk^ypist Centroid 

Number of Activity Preferences 
Outdoor Convivial 



35 T. 



30 T 




(Clerk^Typist 
Centroid ) 



(Tom Basic) 



0 



77 
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Xi - 



12.5882 
12.6882 
12.6882 



24J235 
24J235 
24J235 



12.6882 242235 



p - 1 



p = 1 



ni 

2 XpiXps 
P - 1 

ni , 
P = 1 



, 0 



V86 



P(Xl,X2) = 



20.006920 „ . 4.602029 

4,562629 . 1^8.673504 

D, = NrKX'iXi - X'iXi)Nr' 

1 '^ui^^k 



(85 rows) [Eq. 1]. 



[Eq. 2] 



2irffio'2(l - p')* 



e - >4 



[Eq. 3] 



[Eq. 4] 
[Eq. 8] 
[Eq. 6] 
[Eq.Tl 



Xi « II (X, - X,) (Xj - X,) 

' xiDr'x'i = x» 



[Eq. 10] 
lEq. U) 



7^ 



ERIC 



70 



1953 INVITATIONAL GONFERENGE 



Vigiue 7 

Cartesian Representetion; 
Outdoor and Convivial Activity Preferences— 
Clerk-Typist Centour Specialty Fsychograplw 

Number of Activity Preferences 
5*121 



35 



30-- 



25-- 



« 20 

I 

•'5 
s 
o 

"15 



10 



^112 




+ 



10 15 

Outdoor 



20 



25 




30 
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Table 2 - 

Centour Scores I 
putdoor and Convivial Activity Preferences- 
' . -ClerhTypists 



* Centour < 0.5 



Convivial 


0 


6 


10 


Outdoor 
15 


2Q' 


• 25 


30 


35 


m ' ■ 




02 


04 ' 


02 


* 




30*'- 


* 


05 


' 27 


40 # 


16 


02 




25 


01 


20 ' 


80 


86 . 


26 


02 




20' 


02 


20 


68 


46 


09 


01 




16 


01 . 


05 


10 


06 


di 




ill 


10 


>ii 


* 






* 


* 




6 




* 


* 


* 








0 ' 








* 









Figure 8 

Matrix Representation: 
Outdoor and Conviylal Activity Preferences- 
A^E Mechanics 



20 
21 
16 



19 



27 
15 
27 



16 



(93 rows) ' 
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Figure 9 



. Cartesian Representation I 
Outdoor and Convivial Activity Pref erf nces- 
\Clerk-Typist& and AirE Mechat^^'- 

HmtbT of Activity Prefarette^a 



^112 




• Glerk-Typist 
^ A&E Meehanic 



10 



15 

Outdoo^ 



SO 



as 



' -7 



30 



X, = 
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'18.5376 „ 
18.6379' 

'18.5876*.; * * V- . 



21.1398 
21.1398 
21.1398 



18.6878^-j\^ '21.1398 



/93 
0 



■ 0^ 
V93 



X, = 11 (X, - X,) (X, - Xs) 
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[Eq. 12] 

[Eq. 131 

iEq. U\ 

(Eq. 15] 
[Ec^. 16] 

tEq. m 



,Conuivial 

35 

30. 

25 

20 

15 

10 

5 
0 



Table 3 

Centour Scores i ^ 
Outdoor and Convivial Activity Preferences 
Clerk-TtjpUts and Aiy^^E Mechanics ? 



'peciality 




■ J 




Outdoor 










0 


5 


. 10 


15 


20; 


25 


30 


C-T 




* 




04 


02 


^ " 




A&E 








01 


01 




* 


' 0-T 


* 


27 


40 


10 


02 ' 


'■ * 


A&E 




01 


^ 11 


12 


02 ^ 




0-T 


01 


,20 


80 


86 ^ 


25 


02 


* 


A&E 


* 




06 




61 


ii 




0-T 






68 


45 ' 


09 


01 




A&E 






05 


67 ' 


90 


IP 


' 01 


0-T 






10^ 


. 06 


01 


:* 




A&E 






02 


21 


38 


69 , 


- 4! 


0-T 














* 


A&E 








02 


05 


01 




0-T 








* 




4! 


* 


A&E 




41 


■ ■ * 






* 


* 


0-T 


m 










4! 


A&E • 




m 




* 









* Gentptir < 0.6 



' Flgiw^ 10 

Gafteaian Rapresentationi 
Gmtour S}^dalty Psychographs— 



liumher of Activity Preferences 



3 5 -r 



30.- 



20 ^ . 



^ . y 15 



5-- 





Clerk-Typist 
A&E Mechanic 
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10 ' ' U5 - 
Outdoor 
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summXry of discussion 




... ft/ J 

L Comments on Dr, An^halters pa- 
per. ^ ^ 

It was pointed out that in Dr* Ander- 
r' s c^se a multiple correlation andl- 
sing iikm same final, alaven B^tm 
wouldi havev given ^uivalent r^dlts, - 
i.^^p tile resiJt is obtained by combmlng 
a number of effeets diat are ii^ifieant 
separately at only the 10-15% levteL 
Dx* Thomdllce pointed out that since 
wide-i^read uppCT and lower criterion 
groups had been used to obtain cross- 
vaUJEes in tfie multiple discriminant 
analysii, tiie obtained figure ^of 0.61, 
0f eii.s' would be expected to be lower 
for aviotal population. However, this 
^would not ^affect statistical signiflcanqe. 

Dr, Alman wondtted about the 
effect of skewness in Rorschach scores 
upon tiie. discriminant analysis. Dr, 
Anderhalter indicated tiiat the most 
'^ghly skewed variables had l^een 
eliminated because of their unreh- 
abihty^ and Uiat none were left .among 
the final eleven used. • ^ 

ll pQmmeMs on Dr. TMemun& pa- 

Dr. Solomon suggasted the mul- 
tivariate normal disWbubon is^ in many 
cas^, quite adequate for psychological* 
data^ from the staflstifal point of view* 
He asked Dr, pronbaeh whether any 
conventional terf of significance of the 
disttibutiort is used in the Cronbaeh' 
Glei^r te^nique. \ 

Dt, Cronbach indicated that he saw 
A ^ ' ■ 



no baste disa^eement between Ae 
Cronbich-Gleser D and tfie Mahalano- 
bis Dj except as purposes, ^ere are 
really three distinct types of probl^ 
~(a) determining die similarity of 
^0 individuals to each oflierj (b) de- 
termining die > resemblance i of Individ^ 
uals to a single ^oup (e^g^ Ander- ' 
haltCTi study), and (c) ms^essing the 
relative resemblance of on^ individual 
to two or more \poups (e,g.i Tiede- 
man*s study). D is better for a and 6j 
while JD is Hitter for' iBie point ii 
thatp within a single population, it is 
not always reasonable to reg^d persons 
with equal centbur scores— Isofreaks" 
— ^as effectively equivalent. This can 
be seen by considering a situation in 
which two discriminant fimctions are 
highly correlate^, within a particular 
criterion grouping. A report is due to 
appear the Psychological Bfletin 
(1953, ^ 456-473). ^ ^ 

Dr* Cronbach also suggested that we 
may do ourselves harm by considering 
people who have , modal discriminant 
scores as best by definition; Jthe b^t 
person may actually be one ^^lo devi- 
ate from die mod^ in an advantagedus 
fashion* For examplej die best veteri-' 
.. n^y doetor may have certain quaUti^ 
that are more like regular physidans 
than like the (ypical veterinary, and 
receive a lower eentour score as a vet- 
erinary. The best psychometaicians and 
clinicians may be more like one ano^er 
than the separations of the mod^ of 




;«/ ' TEST! 

diiir two dliMbutiotiii biJ^ o^^^- 
ent ielection procedur^i would^ Itnli'^ 
aat@« Some #6ah^ue is nfadpB fdr 
intepating iart of ^ CKsngld^ation 
into our thinjdrig ateut profile* * 

ppt TiedMian indi^tad that he and 
Df« Rulon have a report pending that 
wiU tponsid^ these and oth^r tQMmB^ 
He sugg^ted that the audience ihoidd 
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e^eriment themielves with the differ- 
ent indices proposfedj si a means of 
discovering the inQoni^tenoies which 
appear as to which people are classic 
fled together* 

The discussion had to be stopped 
befpre the issu^ raised coidd be 
further e^lored. 



it"'--; 
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MaMng Test Results, Mfeanm|fiil 



RALPH F. BERDIE 



BRINGING NATIONAL AIjD RE^i^AL^eSTINgV 
» PROGRAMS INTO LdCA^^DOLS 



MoffT TOsm havA muitipi© us^^ When 
teat scores jtf© used for qsUege admis- 
sion puipcii^ and not used for pur- 
pose wunseltogt taport^t ii^^, 
mation is wastea. When objective "Mt 
data are used for puippiw of evalup 
atog instructioir but not used for pur-i 
posw of pifoviding academle guidance^ 
to sttidants, again mooniplete use ii 
made of &e tests. Tta caneapt of 
"^peAentage effleiency of use" can be 
used in testing to indi^te die Be^ee to 
which ma^dmum use is mftde of test re- 



means of A^e propams ean'ba inte- 
pated with^tiHiction and counieling. 

The extant and coverage of natipnal 
and retfonal toting progfams have ex- 
pand^ since &k M World Ww until 
now ^e^ constitute a.'signifl^mt ac- 
tivity in both education and psyShbl- 
ogy, A numb^ of •national programs 
have dwel^i^d in <r8sponse to needs 
^.^ressed profassional schools and 
^rQl^sional asi6ciati9ns. National test- 
ing*^propams relate^ to prgfesiional 
ttainii^ are foftnd no*^ in Madic^, 



S -Sie purposes of flthls paper are * Dentistry. AccoUnmg. Law Engmeer- 
ISem* dm to tacreas- tagr Nursing. Jo^aUsn* m^^' 



to ^miis problemtv aritag in ingeas- 
ing til© effleiOTt use*^ of tests aAnin- 
Mared M large-scale prcp-ams and to 
discuss means of inereaiing the effl- 
eiency of me, 4 
* National and regio^I testing pro- 
ffmm have developed in |espons^to a 
v^ety of ^ueationd n&ds. Usually 
these prolraras htva beafi conceive| as 
means for^solving wtain broad educa- 
tional problOTis, pccasi^ally &ime 



PharmMy, aid Educatonf Even now, 



attempts are being m^pfe otfier pro- 
fassionij^ to. develop sihiifcr progranis. 
ThousSd^^ students each year are 
t&ed in th'&e propams. _ i 
t The early history of piychomeWos 
iuggesfed tfiat national and re^onal 
fating progran?! would develop in re- 
sponse to demands for. vocational guid- 
%ice aids. Only one tege scale national 



BTobl^ have *aen rathw specif in yo^afli, ftat sponsor|d by *^ Umt^ 
^ Ke^d whef this was &e case, the ftat^ Employment Service has da^ 
Sl^pam has Mri tfie potential veloped in that.direcfaon but ^ de^ 



"Kmewhat p^ter tfian A© iolution of 
die posed problOTt In |very case, 
howt^, vm niust face the qua^tions 
of how ^mm t^tin^roerlms fit to to 
broate tducational picture md 
How 4© dato ofttoindl toDU^ ttie 



ing General Apti^de Test Battery diat 
eventually may be used widi thousand^^ 
of oitf young people* 

Odier testing programs have d^ 
valbped in respOMe to n^4s for ^e 
appraisal of educational efforter The 
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Sophdmorfe Gidbr©' testing pfOg 
the Graduata ' Rf oords E^gaminatiori, 
and A© toting propams in Stota 
of Iowa all have been davattpid; as a 
meaiii to help iducatorf eyaluafl^ ttie . 
rasiJts^f Aair laborti^ 

Somewhat ralatrf' td' the propami 
coming ftijm the protoiional schools ^ % 
are ^ propams deyaloped to help \ learning 
^ co neg ^ Ijaq^fi^ fl T?^ i ^f^"^'^ qualf^e^ doselv rglatc, 
stud^ts^ The pro^am^ of tha College pu^Oies Tnvoivi 
Ehkanqe Examinatioa Boards includi^i 
the QoBeg© ttansfer t^* aW thetpro- 
^ami^bf die AsiQoiation of Minnesota 
CoUegfc have had unique develop- 
ments.^ More recently another type of 
national testing propam Has appeared 
^t'^resptwe to problems raiiad through 
tiie comjguity of the National Defense 
Pr®^am with our inititutiojfe . of higher 
r learning.' The NROTC and, the Selee- 
tive Service ExaniinationB provide im- 
portant coordinating links be^^een boj- 
leges and the defense deparbnent. 

Another variety of a regional testing 
pr^grasd^^has appeared^ a variety that 
has puf^i^ overlapping in large part 
with thii pHrpos^ of many of the pro- 
gfams./jtirft mentioned. These are Uie 
state-\V^^ testing pro-ams now found 
in abdiit forty states ^ programs spon- 
sored by '.gollsges and, universities^ edu- 
^tional%s§oe^tions, state departmenti 

pf thesf . 

ibu- 
the^ 



pWis are utilized for what; might' 
be firahkly called *'|ecruiting" piiiposes, 
IiJ' these cases the purposes of the pro-^ 
gram involve the ideritiflcation of quali- 
fled students witfi the expectotion that ! 
suitable action can tii^ be titken to in% 
cr^se the probability of die.itud^t's 
entemg t^^^ppr^^riatq ihstitution Idi 




eoi 

schools, 
eonsiffii 
differ 
recogt^J] 
sion 0^ 
grams, 

L^rgl 
ibertafet*ai 





adiiiiision bf^ studenfti ' 
and professional sdlbofi 
curricula in mdst sc 
for a relatively homogi 
indi^duals and schooli fre 
lume th^ respbhii^iltty for dp^; 
that all students fttoitted c^ 
rfasonable expectation of dbin^* 
^rk. ^ , \ \ \ 

manyi? of these prog 
eir puipdsa the providii] 
brmatibn to be. use 
dividual studenti^ 
elsj the iinpor^anfg 
phological indiv 
recp^fted atid 
'Aulted in<an ^^at!^ 
' find giiidanp€f^f^i f 

jipprograms p^|da 
fflpt ^o^fimsVof 'a 
^ ^^M^^ttl ftatur©'*pannStf' , 

provide^ li^^s^^t r^gnjze ^om^fof ^ 
the advantag^s.^In^' jgj^^^^^ inyolft^ihg 
the testfeg jpf ^ thouifnds or perhaps. ^ 
hundreds; pf^lSh^sancfe of indiyiB^ls, 
many resbii^ceif; * Wth personnel >ahd , 
fina^ieial,. are wdflab^e to h$- uned in 
the development pf Heijir-tfec^rucal'i^^ 



of educayphi^Sid combinatio] 
agenci^. IHiese several pr< 
volv^ flia testing of hundred 
sands of pupils each yearj 
elemahtary level to the coll< 
and tiiey have result^^in Ir 
objective test infonnatidn poi^SA; 
our schools* i s • 

Obviously, national mdk) r^^^= ^^.thrdu^i ' the jH^. _ . . 
programs serve a large numbtrj^ p^fo^ {Resting $eryi&^a jFail tb=be convinced 
poses. Sdme pro-ams aim Itiy j^itmcting a good testj .^J- 
information to die sdiools t^^^bj^ u|ed '-^^Sgh it may not be neeasiaiV, it der-^ 
primarily for ii^|ip|pionaV' p^ have ayailablf exptefi- 

sotnetim^ for t|9^ppose i^ji^^liid-^ writera, - toafin^^^itprsj 

ualizijag ins^^^^^d sbrneiMiS fby ■BattA'iis piF^ machin^j^^d hi^ 
the ^^postf^^^^^aWing Ins^ti^^ powered itatistibipSC \ -^^ ^ . 
Some of th^l^^nii and rf^onal Another adv^Waga : resulting- from 



stfuirterits. No ^^^^whQ h^^.g&W 
^shoo Irff thb EduiCTOOnat 
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;^"itori^Svifflabte to t^t itandardiza= 
tij^i^^P^I^'a^d for tf(© thousaridi of 
I ^^^phf v^Jo Can provide .normative 

} V A^tiifrd adyahtage derive from mm 
WxleHiive experience poisible to^accu- 
miilate with a teit used in many situa- 
tibns. A t^t uied in hundreda of 




Ralph ^hd^ Taylor intfAedioineJand of 
Weiss and'of Liyton in dentistry "dem» 
onstrate the heied for resetoV 

The . natli^te off tJ>i sponsorship \i of , 
these ^pro^grajps ' alrf" mhy ' have 
effect S-^enoouralinf isicritidal aicepU-^ 



anee on the gart of* t^j|loM schools. 
ut^p|yeh( 

ingj iSj^a^. tq accept witliojit^qjiiestiOT 



A dean-^ a^, school^f pentis 



Ung very jttlp dbouPp|ychological test= 

^the st^pjp 
lean Dental^ 



any batter^ of^tesfe bearir 
of appfovaU oft t^^e Amii 
Asiociatian:, q^ite apArf f rpnjjhe rfteriti 
of the te& then^ielves.^Sininarly,^^ ' 
school tejctiers* rnay fedti rtiat the 
of a university^attaohld to ^6 t^ing 
program tipay ^rv^i.4c^ W^ante#thif 
quality of tbe ptodm^ wit^w s^ously > 
opnddertng th6^^iiatiw of tfe ^mram 
as it relate/ to jhe needs offthe ^Ool , 

In gen^at ]^ suppose v^b^n say that 
one of the; chief 4isa3vamjgelpf the^ 
large scale rmtiofi^J ^and regional te^ 
ing prbgrams k tA they may tend to 

discourad^ tiiW© ^4 ^^^^^ 
meetihg^uoatidlP> problems ^here 
tests have a role to pfayi 

t^ges ana 



Aind of eyid AioeV schooli atill ten^ | 
make usfe of the natioij^l nont^s « 
ported in i^<larg6 s^le progtyms^^ 
they raitel5|%vel# fl^pir 4oc^:jwrt|iip^ -tl 
; Along with thf posSibiUtyJK^ 
-^ip;agiiig' the ^deVelopment w Ipeal 
tests and pdnnsj thtese lafge^i^ftle t^jE^ 
ing praams also aapourage \ 

locally Ip^b^pre^ rand * cdhduit^ 

■by doiens of individuals able to dS-* " these tesM^ prtgramsr are' .afreet^*. 
? velop hypotheses, test theie hypoA The recently published res^cfe' M 
' and attempt to verify tiie expOTfences 

reported by : ofliers. Not oiily is , mC 

rang© 6f* research extended, but also 

ttii^ experiences obtained Uirough the 
- application of testing programs ^ ^e 
. different aM vart^? Ftolly, the wtolt 
; activify of psydhological testing may 

gain ^ough the ^ from" 

fliese Jarg^ ^cale pro^am^ ' ; 
V Now WvUs: Iq^ at possible disa^- 
/ ^ant^g^ 6f riatiohai and regional pro- 

^amt hw cpmlpared to local pro-ams* 

In A iyay* ^V^^y one of the advantages 
[ ' Just 'liVtel alio can be a disadvantage. 
^ ]Rrt;tb^ paft, itlm^ been all toc^ easy for 
, (hbSe ' reaponsibb for national Or re- 
; gipnal ' progf arns-^ to : ign^e spe^fiq 

iii^ds' ind g^als of local test users arid. 
V^ttrisv^ri: sbn\©*cas^^ fesulted in a 

iiilniirii2^ngvbf contributions that could 

'^/^i^^ ty' .^e^pns acquainted with 
^ Ibial^ituatiorit. jPor instanbep a riiediical 
; syipcif ^at (dfecide? to make..use of the 

Jt^tS Available natidnally may / fail . to 

give support to its local tesung pepple 
' who might be in a favoFahje position to 
; develop superior tests fot^he ^elecfion 

of medical studentg^ X - 

;The developmfnV of large scale pro- 
-ams flsaAas apparently discouraged 

■ thii usb of local hontts: Reports pub- 

i&hed'by many persons responsible Wr 

mtional programs demonstrate with 

Mtle /Question that significant difFer- 

ences are found among institutions, in 

some cas^ Uie top students in some 

.^oups are , no better than the bottom 

students in others. In the face of this 



pfay 

Now obviously 'ffiei^ at 
disadvantages ara ngt inevitably 
butbs of large sg^e testing^ pro^f^s 
Many factors d^fflifiine wh^her these 
tests are used eS|ttvely or not* 
can sdent^^piome^f jhe conditions 
conducive ^the effectiiyor ineffe^e 
us^ of tests;^ ' ^, * 1^ 

First, let us look^ th^lwari^n 
of the persdns^tng ^citestsffin only 
a minority of s^oo& are there pfrsoiis 
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him^ tainad to i)S6 ^sychologidai 
lupadonal t^te and only oiir l^ger 
id^ol iysttos hava a perion who may 

^Se^ in ffiy^onifetcte.: In irtosi schqoh^ 
^t^^ii^p yied pi^aifly ttachers 
"C^&dii in gwieraL tj^Ufm are not par- 
^ ^nikWy; waU pr^l^ to nit tots. 
' . iisianqe, yi^^ United States 

Oige^ ISAj ^riM On Cwti^ 



%.,fieation Requirem^iife for Schp^Pfr- 
v^^el inrme IJmted States^' reports 
^at only ^anty-seven " stal^ iyen 
^^toly mention t^ts^nd me^ureinenti * 
in Tiilation to teadi^r wMoation re- 
^iulremants. For alairieritary taaeharSj, 
in Otily twti staim are wurses in tgsts 
and maasoramants r^uirad 4nd in ik 
itat^ such courses ^re optib^al. Eor 
secondary taachersi in thr^e states sndh 
couxsaa are required and in eight 0ktm 
they are optionaL Wpellner an^ WSSd* 
in thair ^'Requirements for Cartifleaybn 
of TeachirsV Counselors, Librarians- 
Administo^ators/* report that in only^sis 
states are such courses requirad ftr 
adbnninistrators and irt seven states sudi 
course are optioriaL For counselors, 
only eight stat^ specify a testing 
course as a requfremant/ These figures 
iDAnnot give an accurate pietura of the 
situation, but certainly they do suggest 
that Gaining in testing is not perceived 
as, a major requirement for teachers in 
most states* We must recognize that in 
our schools are relatively fevi^ persons 
well frained in the use of tests. 

Another condition conducive to the 
ineffective use of tests is the limitations 
of the teste used in programs. Some- 
times th^e tests have poorly demons 
strated validity, if any at all Occasion- 
allyj inadequate norms are used and 
more oftaUj although .^e norms may 
be ad&iuata, ihadequate descriptions 
"bf the norms are provided. Sometimes 
th^ t^ts selected are not; appropriate 
for\tha purpose for which they are pre- 
nted« 

Another condition related to the in- 




eff^Hve ust of t(^ti invdh^a^ tha limi- 
tations of the interpretativa matrtals 
and aids available to those who iise the 
tits, Spmetimei the deicriptive ma- 
teriaU and test manuals are not legible 
or provide ambiguous presentations, 
.Frequently die material prasantad is 
not .^mpletri, occaaionally such Impor^ 
tahr inftnnatiqn as Ah^ concemmg 
validity is ; om ttt^^ ; |i:equ antly illus^ 
:^atlva material u W^ to Help 

; teaeheri and counselors make batter 
lisa^f test scores, 

ENcquriaoiNG AND Imfroving 
, . Local Use of Tests 



* "Undoubte^yi while, ^inj to in- 
crease our awareness of soma of flia 
dangers accompanying the continuing 
development of large scale testing pro- 
grams, I have presented a somewhat 
p^simistic picture, I think, however, 
that the vWed development of these 
programs has demonstrated adequately 
that in the Iqjg run they will benefit 
all of^us, but both we who are ra= 
sponsible for the administratipn of some 
of these prop'ams and the persons who 
are directly using the obtained results 
can do much to increase the efflciant 
use of test scores. 

First, those directly responsible for 
such programs can continually arouse 
and sustain interest in the programs 
and convince persons that if tfiey ac- 
quire the necessary skills, tfie programs 
provide data that can be of unlimited 
value. Wa have tfie responsibility \for 
repeatedly clarifying the purposes of 
these programs. Recognizing the ^eat 
turn-over in educational personnel, toS 
state our purposes only once is not 
enough. Then wa must recognize diat 
many, if not all, of these regional and 
national programs can serve a variety 
of purposes,, not only a single pmpose. 
Although wa may be concerned with 
. one piupose, we must be aware that 
test users will be making additional 
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ules of th^e Mts and we have a re- 
sponiibility for helptog Aem« Thii oalli 
for effeddve and oontin^g communi- 
cation between those who are admins 
^tering Ae programs and the test 
uim. Copper ativie planning and ra^ 
view ^ among Oi^e prions is essential 
and only in fioB vmy can the necessary 
inteFeit be pa^tain^. 
> N^^.o ^y must intiti^t be mous^ 



time or anodiA ar^students iii teachers" 
coUegea or colleges oi education and 
in tfiese coU^es courses in t^^g 
usually are >'^ght, These courses 
should includo infgimation about tastr 



ing pro^ainSp 



and, si^tainedj but relevant information 
.must be pro\dded to tfiose who are 
'lising the t^ti. Hie p^onj sponsible 
for me adminiitration of dhe proems 
mus^ttake avaOfibl^ adequate manuals 
aiid bidl^tir^ d^^bing ,the purposes 
of / tfie programs/ the tests included^ 
possible uB^ at tm% and ^ustra^ 
tive materiais to serve as training aids. 
A eonstant flow of research publica- 
tions should be * maintained between 
those administ^ing die prop^am and ^ 
the test users, research done by the 
central organiEatiorij and by other per- 
sons, Periodically^ seviews of the cur- 
rent status of tiie program should ap- 
pear. Summari^ of tha research 
completed, summaries of clinical ex- 
periences, and^ carefully considered 
Judgments of informed authorities 
should be, communicated to test users, 
Next, the periodic assembling of test 
users provide an opportunity for re- 
viewing problems int the application of 
tests and allows for diose administer-- 
ing the programs to maintain awareness 
of the revealed needs, Similar purposes 
frequently are accomplished by having 
consultants visit test users, perhaps ac- 
tually to observe the operations during 
which the tests are used. In tiie same 
way, the effective use of correspond- 
ence between pro-am adminisfrators 
and t^t users accomplishes these pur- 
poses. 

Relevant formation frequently .can 
be provided to prospBctim telt users 
before tley be^ die Job, particularly 
in programs involvtog eltoentary and 
secondary schoolSi Tmt' users at onf 



Proprani:4'^adminisfrato|S- |requendv , 
e more diective use ofl 
ting users to collect data, 
ge scale research and faff 



can en^^ 
tests by^ 
both foi 



local reiZarch putt^ses, Cooperative 
research frequentl^an be encomraged 
among the vafious schools making use 
of tests provided in a pro|^am. Com- 
parative vahdities can be determined, 
new tests and techniques expOTmented 
with, and interpretive aids bled outi 
Local resear^ projects can be ^eour- 
aged by test administrators^ program 
admihistoators might even encom'age 
research m local settings by providing 
personnel and flnancial .assistance and 
by fapilitating the distributiori of the 
results of such local research. 

Thus, muny things can be done to 
increase the effectiveness of the use of 
these national and regipnal testing pro- 
grams. Most of my attention ^here has 
been given to things that, the program 
administrator can do. Naturally, he 
does not bear all of the responsibility^— 
the test user himself, must share in this 
responsibility, but the program admin- 
istrator usually is the person who must 
assume the initiative. 

In conclusion, I would hke to draw 
these generalizations concerning na- 
tional and^regionat testing programs as 
they relate to local ^useii. First, these, 
prdgrams currently provide a bremen= 
dous'amo^t of useful, objective data 
to lo9al schools. More data is sy^pphed 
at present than the test users are cap- 
able of .using, Next, the presentation of 
test scores |can be improved by pro- 
grtm admihisbttdrs, first by gaining 
jnore pers^pf involvement on the part 
of the test users, aiH secondly, by pro- 
viding therh Wjdi n^fe fraimng: Next, 
more resear«^i| ne^sary if test users 
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and prop'am adminiiteatoi^ are to have 
die mfonnation needed fo mdce^ ade^ 
quato use of datat Next,,, as with 
prohlenii related to thlsj more 
ImagiAatlve developmbiit «f techniqi^s 
and me&odj are needidp* particularly 
on die part of tb0^ program admmiiti'a- 



tors, FinaUy, we must recognize diat 
these toting programs do not exist in 
a vacuum and that iiltimately the effec^ 
tive uie of data obtained thrpugh these 
pro-ams depends upon the improve- 
ment of boA . counsehng and instruC' 
tional pro-ams in our schools. 
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Making Test Results Meanm^l 



MAX D. EN GEL HART 



LOCAL TEST CONSTRUCTION AND'ANALYSIS OF TEST DATA 



Pabtigs^ahon. m itate^ r^onal, or 
national tating propams or me of 
standardized t^ts in local toting pro^ 
gi;ains have certein definite valu^ in 
sfieistminenf and evaluation. It saim 
evidient to the sp^^, howevCT, &at 
loc^ t^t conabruotidD Ib abo an impor^ 
tahrm^ms of maldtig tettag and t^t 
fdbl^ m^min^I to teaches. Through 
p^dpation in stat@p regional^ or na- 
.tional pro^ami and fcough die use of 
steidardized tes^, date may be col- 
lected whidi make pb^ible Mmparisons 
of pupil or student aptitude and attain- 
ments in ^6 local situation with the 
aptitude and attamments of pupils or 
stud^^ taL. other s^^^^ or 
higher instituUqns. Local t^t construe- 
Hon can, howevtt', be a means of pro- 
viding evaluation insteumehti more 
elc^ely related to loc^ instructional ob-* 
Jectives* It can be a means of stimjat- 
^g peatm^ inter^t in toting and in 
use of test restJte by t^diers. Teachers 
a^erienced in t^ consttuction are 
uiuidly bttter able to select standard^ 
ized t^ts for local use and to under- 
itand data ^Uect^ bi gtetep regional, 
or nation^ programs* Just as an ama- 
teur artist or musician may have a bet- 
ter unda^tanding and appr^iation of 
^e works of prof^ional artiste or 
m^idan^ Aan the persoQ who has 
nev^ tted to paint or play, the teadho' 
^oi^ted in tM ^^^^ction may 
have a gr^t^ unde^tending and ap^ 



pr^ation ^ of ^ professionany prepared 
im&. ^ . ^ 

The test expert working isdA teach- 
ei^ in local test construction encountws 
a variety of problems. T^Aera difftf 
widrty in their - knowl^gV of t^ng 
and' Aeir attitudes toward it. Where a 
^oup of teaA^s are'^ked to coop^- 
ate in tiie production of an examination 
to be given at tiie end of tiie come 
which all W tiie teach^ bib teadiingi ' 
the members of the group can be eac- 
pected to react in a variety of ways. 
Some members of the gfo^p de- 
plore the idea of preparing objective 
exerelseSi Other members, if askrf f6r 
contributions of ex^^^, will prepare 
series of "^frue-ifelae pr rnultipl^answ^ 
exercises relevant to facte of importance 

, only to themielves. Often sudi exercise 
are contributed by the teadiers who 

' would have prtefOT^ ^say quesUons, 
or who contend that attainment of 
wortiiwhile . objectives can only be 
measured by* essay questions* fte- 
quently, however, tiie t^t expM is 

' pleasantly surprised by toding one or 
more teachers in the ^oup who have a 
considerable understandirig of what 
should be done and ability in prepi^g 
effective exercises. 

The logical first step in wnsfructing 
m ' achievement test by a ^up of 
teachm is to^ have the teaehOT define 
thek instructional objective* It has 
be^ tiie ^cperience of tiie it 
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howevw, that to wkingsteach- 
to Hit tht^ inifruc^nal objaativ^ 
ialdom r^ults in a ' usdFul con|pilation 
of dbj^tiv^* Adequate dfefiffltton of 
objective devdoptf ilbwty and tiirou^ 
continued odop^ati've el^^rts; in t^t 
oonstaiiotion and in anal^sij|tf t<M, data, 
in planning Ae eons^ction of a t^t 
vdtfi a group of .t^chlls^ ij^tiating teat 
coniteuctionp discussioriy 6f objeotivas 
and of ga^erd^fis 



att^mnent of tfie objective needs to be 
quit©' gaEteral md demeutary* It is 
helpful to provid© tiie teacher^ vrftfi 
example of various typ^ of objective 
^^oiaai and wi& npt^ .^qjlaining nbw 
to write such axeralies. X^mh notm 
may call attention, to iuah ffiirtp as 
having the problem of a mijtiple-aniw^ 
exerdise set in Ae stem of the exarcisep 
having the answer of parallel constfuo- 
tion, haying all of the disWetori plausi- 
hhf and avolduag the mak^g of tiie 
correct answer ie longest and most in- 
volved onep It is desirabla to attempt 
some deftaiUon of objective. It is im- 
portant on the level of knowledge and 
understandings to identify the common 
"content of instruction other\^e many 
exercii^ wiU be contributed by indi- 
vidual teachers which the group as' a 
whole will later reject In initiating co-^^ 
operative test consbuctioa one cannofe 
obtain tiie emphasis on production of 
^ercises designed to measure intellec- 
tual skiUs that ^n be reached after 
saveril semestCTs of exMrience, As a 
matter of fact, ^phasii^n such exer- 
cisas in the earli^t tests prepared may 
be quite unjustified since such skills 
are not likely to be among the real ob- 
jectives of insttuction. One of the major 
advantage of locals test construction is 
_^at it can contribute to \yidening the 
sco^ of instcuctional objectives to to- 
elude such skills* It seems worthwhile, 
however^ eveh when initiating local 
test consfructioni to seek some depart- 
ure from evaluy||n of fao^ or Monna- 
tion illone, Ft^B^s ha$ bee^ made; 




when teachers accept thf idea that an 
oj^jective , exercise will measure more 
tfian factual knowledge when the exer- 
cise represents to some extent a prob- 
lem situation and require applieatioh 
of knowledge in context oQier tiian that 
in which die knowledge waSjtaught. ^ 
After a group of teachers has met for 
the pu^ose of planning an ecanunation 
and various imembers of th^ poup have 
ao^ ptfe^ i^ PQj^biUd^ fa the witing- 
of ^ercAs, it is d^irable for the test \ 
expert to seek consultations with tiif?, 
individual teachers, Such conferenct^*W 
may .be an effective device in stimulat- 
ing the tea^chers to begirt work prior to 
the deadline set. Where a t^chWr has 
written somCs or even aU ^ his exer- 

^^ses, tile conference can ba a means of 
giving the teadier helpful advice widb ^ 
respect to exercise writing techniques. 
Such conference are also an oppbr- * 
tunity for the teit expert to become 
much more familiar with, the real obp^ 
jectives :of insfrucUoUi 

After the ekereises have been pre 
pared it is desirable to have the conbi- 
butions of the various members of 
group evaluate each, others work. Where 
the , group of teachers of a course is so 

' large that exercise w^^ng may be done 
by a commitfeej it is desirable-tQ have 
all tochers participaJte in tiie evalua- 
tion of the exercises. The test expert 
may^ synthesize these evaluations or it 
be done in a later meeting of the 
ffitire group of teachers or of the cbm- 
mittee, It would be ideal, of course^ if 
the exercises finally selected are aU 
approved by all of the teachers. Un- 
fortunately tiiis ideal can seldom be 
realized. It will usually happeUj how- 
ever J that many of thf exercises re- 
jected are judged faulty by most of the 
teachers. This is especially true of exer^ 
cises ^hieh are factually incdireot or 
which pertain to ' content uniquely 
taught by one of the teachers * The op- 
portunity to participle in the evalua- 
yon of exercises is appreciated by 
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. ad ^vfti oj^ftonityi use 
fm not SLpptovM fa^A^Ao or ^ . 
be t^eb^, but AppfllNd by 
^ a WTO^ w© mutih l^s 
hkdy to pnE^te ;r^^tii^tat when Aef 
ti^is^^. ' ^ fir 1 \ 

The phnning , A an e^^^ation, the 
writing^ /^^^fc ;«id]^^th^ mtioal 
. \ ^^Uation 911 take t|m^ Nf^ singld 
: teadi^ shpuld be mkei to eontfibute 

^ ttere should b© plen^ 'of toie for 
. eviduation^ i^ffigj tod dupUoation of 
l^e finish^ t^t* Hen^, It Js desirftble ^ 
^ to inidate the mirk early m a iemester 
and to keep ttie work fno\^g during 

./ It was said above adequate defl-* 
" ^ ntttoa of o^yiv^ req^p and 
? that tfUs is p^eularly ^e of obj^ 
tives pCTt^ning to inteilactual sMIb. 
Th^. foUowing list of suc^ ^bjettlvei 
' was written for use m fte production of 
* social scaenee exardsea pertaining to 
quoted material and as a gttide for in- 
jection^ in die sodal icience general 
" course of die Chicago City Junior Col- 
* lege. The Ust was prepar^ by a social, 
scienee insttuctor* long intereited In 
. evaluation widi die help of odier in- 
structors most of whom have had soma 
yg^g of experience in ^local test cAi- 
i sbnetion. ' / 

The kinds' of intellectual abilities we 
* , should attempt to measure include i 
' 1. The abfiUfy to identify the central 
lisue or^problOTii, 
t 2, iTie ability to recognizi and un- 

^ . derstand underljdng assumptions, 

'. ■ \ : 3* 'Hie ^ abiUty to identify the hy- 
. . ^ pbAesis, or hypodiesea, tested by 
^ . die data prriented in the quoted 
materiaL 

' 4* ^e abirfty^ to s^alyze an armi- 
ment wii^ rppect to bias, 
% , ^^onal fdetors, and propagati^a 

. ,5. "^e abili^ to disdngtiish between 
fji^i opinionSi points o^vie^^* 
and valu^ jud^ents, 



6* Th& ability to analyze the lo^o of 

an,^gw&ent* 
7* lie abihty to evaluate ^ata ot,.^ 
evidence in terms of rde^ance^ to 
^© problem and widi r^peot to 
^ di0fr adequacy, to prove or dis- 
prove' conclusions^ or geiieraliza- 
tions based ypon 4ie data, 
,8. Tlia ability to identify conclusions 
or l^ferences definitely supported 
pmbflbly ^^93rtgd 
by the data, "or conversdys defi- 
nitely disproved, or probably dir- 
proved by the data* 
9, l^e ability to recognize and un? 
' de^tand relationships and aspeot^ 
of similarity^ and difference* 0 
10. The ab^ty to identify le^timate 
ptedigtions of effect? or o£ sooial 
consequences of given comrses of 
action, ; ^ 

The above list was incorporated in a 
statement entitled "Suggestionr for 
Writing 'Exercises' for, Bocidl Science 
Examinations" whidi was mimeo- 
graphed for disWbution to of the 
teachers ^onb'ibuting to the social scl= 
ence examinations* In order to increase 
the effectiveness of the list for exercise 
writing, a briefer^tatement of each ob- 
jectiye was followed by two or t^e^ 
exercises from past examinations Judged 
to be useful in "evaluating the skill de-/ 
fined* Time wiU.'permit the jquotlng 6f 
only ^ a very few examples, AH of th^e. 
exercises are frani stries of exwcises 
pertaining to selections presented in the 
examinations, pr disdributed to die stu- 
dents for ^dy prior ta the giving of 
the examinatioits. It is to be hppcd that 
^ something of tiii nature df the quoted 
material can be Inferred from the exer- 
cises. * ' * 

Identification of central issue or prob- 
lem. . . M ^ 

Bie piragipphs quoted . above dii- 
cuss a condition which is r^ost relevant 



* Hymen^ Chausow of the Wright Branch 
of ^die Chicago City Jiinibr College* 
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Iteo^f goveramentP $ \ 

. A. Whirfi is betters a pwrliamentwy - 

^ ^ or ,presidantial team I of govern^ 
= . mmtf - \ "i •• , * • /* 

. B. WU^ is batti^, a dimoar&tio or 
a' totalitwift^^govttiM^entp 
Grf Whi^ 'fa bett©fj governme^ by 
}^ .mp^^m0t government r^^ting. 
^0 of th© massesf . 
WKich is bett^, a govemfiaint^ 
'%as^ on direot or milir^t dffiLOc^^ 
racyP 

Reoo^tion • aiid ^dwitanding^ of ' 
sumptions.' \ -..^ " ■ 

R^ardte^^e role of government in 
sodety, Ao autibor Appuantly assumes 

? : A* ''^at Govermnent is* best 
ijibv^rxls ImMt^'^ 

*^^h^o should be more buslneis in 
govemnent aild less govmini^nt 
Jn busiiless.'" ■ » 
govemnient should , dfareet ^ eco- 
nomio ao^vi^es for the good of 
tt© jeople* - - " 

govenmient should do for^ di© 
people wh'&t they/ as individualSj 
cannot Ap for thOTisalv^, / 
j|?sis of an argument widi reipect 
to its bgic. ^ / ; t 

Onb ean .make a very good case to ^ 
^ ahow tiiat Mr* Brdwn is infconsist^i^ . 
when he ^ . f i ; 

^ is against expanding gQvernni«^ty 
functions but is fw a St. Liw- 
i',. rence Waterway. 

B. ;is spealdng as a private citizen 
J while at tiha ^aame time speaking \ 
* aa a rf prestatative of tiia John 
- T Jones Company, 

0* W©sj6 show tiiat ^e aims of die 
""^^^AntFTas Gro\^ are compatible 
witji atos of his finni 
^ advocates thf elimination of 
, subsidi^^ but also wants 
^dupe government expendfoires. 
Identification of conclusions^ or infer- 
ence supported or not * supported by 
ft© date. / \ ^ 




Qn ft© bqi$b of evidenc© jjresented, 
by til© au^'or one can logically con- 
clude ^at ♦ 

A^ ernployers are usually right in tii© 
disputes which arii© between 
management and labor. 
B# the laboring class can not be 
pected to know wfcat is b^t for 
society, ^ 
.Cti^tiie laboring class uually demands 
'.■ft at wMK}^~E7^eti'imenta^ to so- V 
' ciety, , J ' 
* P*, die jnethods usrf by the laboring 
^ class to secure their demands are 
sometimes debrimentid to society,^ 
Bfedictipn of Consequences or effecti, 
V If 'tit©, saW tax advpeated' by the 
author of the above quotation w^er tQ . 
b0 adopted, w;hioh of;the followtog 
would be likely to happen? 

At Less money ^yould be collected 

by meari of tiie n|come tax, 
B, ^People with large incomes ' would 
suffCT niOJfe:;than people witii^ 
^ small indomei* 
PeogJe M^ithirihldl income^.. 
. ' suffer jnoretTihan ppopl^* with 
large ineomiBS^ ■ 
D. The burden of tiie tax; wauld be " 
tfe _ proportionate to income , 
V. >The - ©x^cisea Just quoted are aU of 
tHf multiplMho^ce type; Other types 
ireqtteritlv-*^Btf include itemi ^to be 
felassffleg aecording to sui^ categories 
as '^A. if^ the*flrst! speaker would agree, ■ 
' Br if the second I Speaker woul^ apee, 
G. if both speakers woiild a^eej and D. 
if neither speaker would agrigej' ;or '"A. 
definitely fru©, probably bxifj C, in- 
suffleient evidence, D. j^robably false, 
and E. definitely fals'e;^ In oxfy given, 
examination series of exercises iof vari- 
ous types are also/included which 
not refer, to selections of quoted ma- 
teriaL^* j 
\ Many instructor exhibit increasing 
enthusiasm - lor e&rcises refeVant to 
quote4 inaterial feeling tiiat lueh exer" 
cises tninimize recall of information and 
do evaluc^te iinportant intellectual slmls.' 
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O&kwt tea^m, hdWevsr, dej^recate 
such e^eroij^ and ^im "diat s^d^ts 
can answer them suoo^fully without 
B&iom study of^'tibe equips. Apart from ^ 
hoping that iu^ UBtructdrs soon- 
retire, the tmt .^qpert tvorkihgV with 
tesd^OT^^^nn^0 an rffort to^aog^ * 
sucfi ^tfitucfe* It ^ helpful to oall at^ 
tention to faot Aat su^ ^^cisas 
are b^c^mlng intf^singly usad i]:i,^tha ^ 
new^ and batter st^d^di^ed tests and 
-in the t^ts of important regional and 
national .. toting prop'ains^ It is some- 
t^es conVdnoing to point out^diat such 
e^^Qisei nave funQtioni similar to tiie 
^say qu^ons teac^^ has claimed.^ 
nieastffe ^@ worthwhile objective. The, 
argmnent tfiat no study is nectary has 
I^s forice if the selection^^ which 
Meroii^ are bas^ are ^osen wisely, 
niey should be of jufflcient diflBculty 
^at experianc^ gained during instruc- 
tion contribute to understanding of the 
sil^tionSt Gi^ain of the e^ercis^ in 
Ae series whidi follows a selection may 
m^jny e background knowledge of 
terminology^ facte, principl^i or con- 
ditions relevant to the quot^ material^, \ 
but not defined - or explained therein. 
For ^amplei suppose a para^aph on 
'Some labor problem include iuab tenns 
as "dosed shop" and "imion shop'* and' 
&at those termi are fwt defined in tiih 
parap^aph. An ex^cise may concenv 
tifie selection of the b^t statement of 
the difference between the dos«i shop 
and ^e union shopt or *two exercis*^ 
may be written involving separate defi- 
nitions of Aese terms. Suppose 'that a 
paragraph represents a prpposal for the 
adoption of some |otin, of sd^ tax. 
While odier forms of t&cation are ^ot 
dii&uss^^ it wiU be fegitimati to have 
jome of me exercise deal widi the rela- 
tive mwite of dffiierentjdnds of taxation, 
Sudi efforts to ^crease die rdevance 
of the exercisfs to tiie subject matter 
tau^t .definitely make ^le exercise 
more valid wiA f^peet to instruction 
and do tend to promote acceptance of 



#xerdp^ in the sani© series of axferiises 
jnore concerned wit^ %valuation ^of in^i 
telltetual skil|^. 

It is the belief of , the speaker ^at 
while teachers ^ are' engaged ^ in the ao^ 
tivity of exercise writing they a^e de- 
voting ^ou^t'*ta JiistructiDBal objec^ 
tivt^rT^e f^ehSr aslcs himplf wheffier -~ 
the knowledge required iri solying an 
exerc^e is within the* sc^eH-of die 
knowledge taught, the ^ereise is one 
wlych required application of ^ totellee- ^ 
tuaj sftiUs, the teacher may ask himself ' 
whedier ot not most of die students can 
make sudi iipplication of intellectual 
skills. Jn odier words^ the teaser at^ 
tempts to fori^st the student^beHaviors ^ 
and ^ imagine what will go on in the 
mind of the student cbftfronted with th# , 
exercise. Even in* witing disb'al^ors 
die teacher may usie experience gained 
in insixuetion to phrase disttactors 
which owe their effectiven^s to J^ie 
facf that they are the kiftd of answ^^ 
he can expect from less able students, 
s Similar thinking occurs in the mind of 
the teacher cafefuUy evaluating exer- 
cises ^ prepared by his coUeagu^, 
Finallyi^ while p^r^cipating in die se- 
lilction ^ of exercises for the test as a " 
whole, the teacher must thiak about 
whether or hot the^'^^amination wiD 
sample die content of the comrse in a 
representative .mannfer and whether 
there is an appropriate balance with 
jespect to die evaluation of various in- 
^bruetitimal objeoHvas.-^e teach^ who 
has pdrtieipated in thtfwholtfprocess of 
test constonption arid evaluation comes'' 
to accept testing as a^a^r and essen- 
tial f unetiqn of insfructioii. 

The preceding discussion has been 
largely eoncdned widi some of die 
proHeihs of local test consbiiction and 
Fhope has indicated^ at least indirectly, 
how participation in tert constoiction 
can make testing more meaningful to 
teachers. ' '* ^ 

After the tfets have been given and. 
fcored a carefully written r^ort can be 
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a mmm of making &e trtt retulte 
meaningEul to the t^^6rs,. Most in- 
j^ctors undmtand tabl^ pf^^antjQg 
frequency diab^ibutions of iotirm and 
iu^ avmig^ as means and m^ans, 
Standafd deviatiDnSj Itandard BOOTm, 
^^nd piroinffl^ faScs need repiNt^^Sc^ 
planatjqn ^both in report and in dis- 
ci^ion teai^ei^. '^e same is bna 
of c»ejSBcients of coiralation* Under-' 
standing of these fliinp dom^^slpwlyj 
but as sem^teri pass it is Ratifying to 
find more and more teaehei^ interested 
'in iuch data and increasingly famiUar 
with their meaning. Participation in the 
consfaTJction of the tests is surely a fac- 
tor. ; 

In add-on to reporting summaries 
of test scor^, testing and test results 
become more meaningful , to taachers 
if the te^ch^rs a^e given the item analy- 
sis data relevant to the tests to which 
.diey have contaibuted* We enters ad- 
jacent ^^^ach exercisa in several copies 
jof each test, tiie item difflcujty and the 
. item ^ test correlation* The 'item diffl- 
cult^, percents of correct responses 
make it possible for each teacher to 
evaluate achievement ; in terms of in- 
structional^ objectives. Both the item 
difficulties and the itenytest correlations 
are of great help to taachers when re- 
vising a series of exeicises for future 



-use. ^e instfuctojs of one of -our de- 
parbnehts are not satisfl^' with analy- 
sis pertaining to only the conrect »an- 
swers* They have .insisted that revision 
ol axerfcises requji^ knolvledge of the 
proportions of studenti choosmg ea^ 
disfractorV It is evident fiom subsaquent 
item analysis that revisions based on 
such data have materiaDy unproved ^tiia 
exetcisas, ^ . ' . 

Prticipation in cooperative effortt 
in test consfruction is a factor in the 
improvement of insbruction and in the 
improvement of evaluation* The effects 
are not restricted^ howevafj to course 
knd examinations in which the coopera- 
tive effqrt occurs. The informal testing 
dona by the teachers in their other 
courses is affected. The teachers co- 
operadng in the construction of tests ' 
acquire more interest in the testing 
done for guidance and placement pur- 
poses. They become interested in 
measurement of objectives^ other than* 
knowledge and understandings and of 
intellectual skills and want to know 
how to construct or select instruments 
for the measurement of attitudes or 
other taits. They become enthusiastic 
about participation in evaluation studies. 
The examiner's problem finally becomes 
one of keieping up mth the teach^ 
with which he has to work. 
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\ Maki% Test ResiJm^M 



THE TEST M^UAL. AS A MEDIUM Or COMMUNICATION^ 



iict Aar tiiii diseussion of ttie 
tiSt manual ai a msdium of i^mmum- 
^oatiori is pwt of a ptopam devotrf to 
. th%f waral topie ^MaWng Test ReiultJ 1 
M^^gfd" dietetas tiiat wa concern 
purselv^ with problCTis of fransmittal, 
^ test manuab^^of Aosa taowledges 
^ %M sldlls Aat snll enable t^^eri to 
^i^mpr^fend, evaluate, and rn^^ more 
effective me of t^f results^ The inplu- 
: sk>n of the topic "Making Test Reiults 
jVl^anUigftir iti thii wnfara»ce ^e^ays ^ 
' 01^ awftlreness Aat teat r^ults in^many 
f instance^ are lass meaningful .to users 
TV than" Ui^ laight be, and ^ur hope that 
at least :sbm© increase in underitaiiding 
. >can be ! effected ttoough improved test 
miriuaU. Few wUl question the need 
for better understanding of test r^idts 
by ihB user, Durosft' in sunmiarifflng 
trends iin toting a year ago,: noted that 
**Taphnical development| in te^con^ 
I stoi;ctibn hAve oumm practoe and 
th^e ks a widening gulf betv^ean Uia 
. test daker ^a test user Traxlerp 
' raporting on a, recent survey of testing 
practices in large eitiei, states that tiie 
majojf problan in testing is new tiiat of 
, comi^unication* including the communis 
catioh ©if die tet maker's knowledge 
abot^ die^ uses and misuses of appraisal 
insbiiHnents to tiie aetual users of tests,^ 
Ho# soundly b'ased is die hope that we 
shall bfe able to Improve this situation 
, appbeiably by improving tesl manuals 
i3 dnodiOT matter^ and die one with, 
wWch this paper will be largely cdh- 
carhedt ' ^ 

' ■ 9D 



' ^ ^^^^ . . ^ " ^ ^ :^ ■ V > 

In the remarks that fdUoWs I have in 
mind partigiJdrly m'anuaU.fpr^.achi^ve-^ 
hientp intelligence, and ai^titude testSf iy 
' arid when I speak df Aa ''t^t user " I 
am blinking ganeraUy of , the classroom ; 
teacher,;^ die guidance iiounselor, and 
the supervisor, rather Aan of A© psy= ^ 
chblogist, tfie director ol re^aardh or 4e 
test specials* ' . . . 

T^e proBlem of the test manual as a 
meSium of communication has two as- 
pects: what is tp Jje ccnmnunicated, 
and how it may best be commutiicated* 
Let us consider each of these in turn, 

Content of the rmmd. m*st;M> 
tmt manual include? What information 
is necessary In order tiiat test rasulte oe 
meaningful? Thera is substantial agre^ 
ment at least as to tha^ topics that ought 
to be povered, If net as to the detail 
with whidi each should be brdated. Tlus 
agreement has been fairly well sum- 
m^ed m tlfe APA's set of technigal 
recOmmefldatidifefbr^test manuals.^ As 
a minimum, a ^ftiual shoul4 include 
specific dirfiqtions for giving and scor- 
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pti pmfiasi^ and ^ni^ation 

■ Few vidQ deny diaf aUlj^eie Idndj.bf 

to^^i^d C ba^bST undmtanduig tot 

futo^ !oC^ ffit *f«dti/;\^ well 
wpttd^s however^ .wh^w 4e infomja- 
tioh. vw^yf^indudei th^e head- 
mgs %hat 't^^ teachei' or 

coimsalor m&d^ td make th& test rasults 
maan^^ul fo| Um^vFdf fasulta to 
be meanin^^ to a teat 'users' st is neces- 
saiy h%{be abia to see cbnneetioni ■ 
between the' t^t results and problems 
which are real and urgent for him— to 
pgrceive^feow^^e t^t resuto are related 
to tl^^oalsTO^^kic^ he 4s sbriving. In 
my judgment, jnforsiation about item 
validities, mm of meaiurement, selec- / 
tipn of non§r jroupij and thip Ukej does^ 
not help ^6 usef to see thiia connec- 
tions--Hn fajetj.oiay set, up a barriej^to^ 
such undeVstanding by fopusing i^Adu^ 
attention on the teehnieaUftes, ^/ 

When we ask what infonnatipn is " 
necessary fin qrder that test, Results be 
meanin^jul, we must also /isk "Mean« 
ingful fe/r whom?*' For tti^ t^^aeherp for 
the dpfotor of researcK? for the coun- 
selor 7 for the adminiktrator?' Each is 
ig something different in the re- 
each has his own purpose in"" 
I Information that will add to the 
^eaningf ulness of tfie results ^r one ^ 
may not for another*. The effort to meet 
the needs of Aeie various audiences, 
Offering as they do in tiraining^ under^- 
sfanding, and test sophistication, greatly 
complicate Ae task of preparing a' 
satisfactoryc^manualj both from the * 
standpoint of providing suffleiently for 
all tibeir needs, and with respecf to the 

* If it be truej as some feel, diat man- 
uals do not provide as much technical 
iiifqraiatfon as thi specialist rieeds for 
adequate evaluation, of a test, it .is far 



moti eertam ttiat tibey do not pro^af j 
0riQU^ asiiitahce to^^^e /tea&er ^ia/v* 
\^i^@lor in tKe proper iiseJof the t^^.'^ 
Hpiv much: help ;caii.a teaiAier derive 
from the average ; achievement test 



iubh matters as marldng and grading, 
toprbvement ,of insbruotidnr motivation 
df pupils, diagnosis rf^ediationP 
TTie teacher n#eds"more spedific^ r eeonnt- 
mendations, mpre concrete fltuitatiVe 
material than is qrdinaarily provided-en 
these applications of test data,. 

L^t yoijttmisinteipret what ^ am say- 
ing as a recommendation to reduce the 
amount of J technical information ifr 
favor oi the how-to-do?it type of eon- 
tent, let me, hasten to" say, 0iat tiie 
solution certiinly does not lie in the 
direction of providing |ess technical in- 
^4ormation, eyjbn though we miy suspect 
Aat such information goes unread or 
uricomprehebded by the rnajority of 
test users, hui rrfther in- the provision 
of more of the kind of information that 
relates the test results to the tiser^s pwn 
needs and ^i^blrnns* ^ ' - ^ . . . , 

The attempt to provide all the kkids 
of assistance that all types of users ne^ 
has one inevitable consequenfi^^the 
unending e?^ansipn of test maiiunb. 
This- poses a moit .difficult problem, the 
possible f solutions to which we cannot 
even begin to eonsider,,here* Whatever 
the proposed solution— whether it be 
reorganizatiph' of material within the 
manual, relegating technical data to ari 
appendix, publication of extensive man- 
uals not ineliided in tfst packages, or. 
publication of sevwal dten^ tojtake ihm 
p]§LQm of the fraditional manual, aU of . 
which have been resorted to-^it leaves 
something to be d^ired for adequacy 
qf communication, Until, it is safe to 
assume considerably more braining in 
measurement than the typicah teacher 
or counselor possesses, at the present 
time, the need, for more extensive acces- 
sory material- fbr tests than can con- 
veniently be provided will persist* 
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\Iiet us now turn ftom ^eons 
of iol^t the manual ihpuld 
catej to the how of its comm 
Can the ordinary juser tead, 
hand and apply inforniation ai 

inanudls be improved in tibis 

Explored ihe flatter 'of readabilto#i^ 
' > manuals by cqinputing Dale-Chall¥ead*1 

ability indices "for tii^ nianuals for ! a 

group of achieveinant,,intflligeriee, and'* 

aptitude tasti, The tests involved — alll 

widfely tisa^-^are those of five diftrerit 

publishers,*: and I may say that there 

was remai'kably little /variatioij/ among 
/ them iiSthe matt^ of read^bility^JJ)^^ 

UeVe fliese ^fiiuals to be Veaspn^ly 

representativei'The average inanncil was 

found to b&i on the whole, of about' 
/ll^ft ^ IZih gradje 'reading difficulty,; 
'which #ouJd hardly appear' to be 

ypnd the level of the ordinary te^lier.' 

Within any manual readability or con-;| 
. tent y aria? aiipreciably froni one part to 

another, Those actions thai consist of 
r the speeiflc durectfbns for admihist^^^ 

and s'gbring tend to be the easiest; Ahose 

sections that have to do with terfinical 

aspects of a teat— stan^rdization djiia,. 
: reliabihty, and test validity, etcT-^ 

afe^ as we would expect, mora diScult. 

[In this sample of . maniials the more 

Mifflcult sectioiis were' of college gradu- 
\ ate level of reading ^difficulty, according 
* to the Ddl^Oiail 'values. We may 

safely assime that material of this level 

df ^ffleiil^ will be skipped^or inade^ ^ 

quately COTipreherided by substantird would, ther 

pr^rtioris of typical test users. least as h 

Can 



than a reflecfion of jthe literaiy 
it 91 tfie aiithor. Tedihical eon» 
iisually of a quasi-matheinali^l 
act^'i and specialised' vocabulary^ 
and should not be avoided' en* 
r^4n^my^ad^uate^teatofiPt; Q^^^ 
ipment of a test and interprata» ./ 
jf-resultSi "We must be wiling ; 
^ito presuppose enough taiAli^g in 
irement to equip the iser witfi 
late backgrbund for undriitand- jt 
fflg*^is technical infonnatiQn^ or to 
Lpknowledge that this infonnaation ; 

' iuali wiirnot contribute gr^Uy ' to % 
theimeaningfulness of results* 
^ComprehermbUity. The fact that ma- - ' 
teal is readablej of eoursej does not 
^Bftrantee that it will in fact be read 
and' comprehended. In an effort' to ob- . 
fain some insight into ho^ well typical 
tisers actually understand die contents . 
of a test manual, we have reqentl^MSa^^r 
ducted a. study of die extent tcH^Kioh ^^ 
teachers could readj oompreheriB and 
apply information presented in the . 
manual for the n^ ecJiUon of Stanford . 
Ackiev&ment Tmt, We prepared^ a 08« 
item tfst covering information specific 
cally set forth in this manual andifld- 
.ministered it to two groyps j^f tfeachejs 
taking summer-session eqursas in tests . 
and measuremKits. Prior to taking the 
test the examinees ha^ been directed to 
read the manual carefully, as if they 
were going to give the tests; and were 
further told that^ey woujd be jested 
on their kno^^^o of the content of 
the manual. I]peading the manual th^ 
re, probably have been 
ily inotivated as the ordi- 



test manuals be written more / nary teapherlkpreparing to give and 
so that they will comiriunicate^ score the test. * They took the test on 



i 



simj ^ , 

niore effertively to users of relative 
limited measurement background? 
have no doubt that they can— vyiftiin 
limits. The fact that t^e various man- 
uals which we studied were so similar 
iareadabiUty cauies nie^'to suppose that 
the level of readability is prbtty much 
.inherent in the, nature of the material, 



tl|e manual first as a blosed-book exam- 
ination without access to the mandal 
and later with the benefit of the manual 
at hand. <' - ■ ^ 

Tlie results under either . condition 
evinced a disappointing level pf mastery 
of the content of the manuaL The aver^ 
age per cent correct on the items of the 
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Resting 

; ' ti^t unde^ the open-bpc^ dbnctition wub 
a'littie over|6p# Oijyiojit item wai an- 

' spite &e fajbt diatf ffi^ aris^ner/to wtu- 
/ dly eVi^ytt^ wis |r&idily opteinable 

jWe have _ studi^| thi^iuits of fliis 
t^t item by^it^ta prd^ to. get toss- 
as tp dia sdrtm^ of difflpulty , and 
, nature of 'misundfi^tandin^. I ;^ust 
confess that 1 h^ve b^en unable to 
rive at any g6hea'ffltotioiis in which I ' 
would repose much bonfidenea^in f act, 
sOTi.e .of the refulti Lflnd in^cplicmble. 
But i venture at least tiiase few obser- 
vatloni; / ^ 

' /ftg^ Items requiring^ some op^atipn™ 
/ 0,g.i conversion of" sgores to ^ade 
equivalents \pr percentile r rankSp 
calculation of, months abova or be- 
low norm — were harde^ than 
, items requiring merely location of 
infonnation*. This . finding we in- 
terpret as evidence of the limita- 
tions of the printed word as a 
medium for developing skills. 
Verbal, directions for carrying out 
operations that; are > essentially 
qui^ simple frequently may create 
an impression diat th^e opera- 
tions are complicated andrby a^ 
odd paradOKi the more complete^ 
and careful the directions ^ the 
greater the likelihood that they 
will seem hardp An^ qperaUon 
which can be demonsfrated very 
easily may in verbal presentation 
appear distiirbinply complex, This 
fact hi^ilights me importance of 
workshops, teachers" meetings, 
and in-service fraining programs^ 
in I which the operations of admin- 
steringj scoring, and inteipjeting 
t^t results can b$ demonstrated, 
and in vvWch teachers who are to 
be using tests may have an o^or- 
tunity to perform 'the necessary 
' 7 op^ations und^r supervision. It 
furdier suggests the des^abihty of 



PftpBLEMS # ' 

V oi-ftdditipn visualimtiori "J^ 1 
u^ ChemselVes* ^; ' 

' - bi 1 Items requiririg tiie reading bf 
* ' ; tables proyed diflBcult For exam- 
/ ; Vple, an iteni reading **TCe most' 
_:^tolrtgjitttnfee^ 

Intermediate Battery is to be 
^swered by reteeriee tq a simfle 
: table of reliability coeflBclents, was ; 
answered correctly by fewer than ^ 
half the examinees; and odier ' 
table-reading itenis Wire similarly 
Hard.. This indicates the necessity . , 
textiial decription bfi tabular ^ 
\ Sata, and of provision of examples 
of how tables are to be *used*/^ 
vAgaini #Jioi*;-hoti; and supervised 
practice seerti called forV 
c. Sonie it0ms answered almost; ver= 
balim in the text of thi^manual 
were missed by surprisingly large 
"percentages of the subjects^ To 
illustralej consider the itemi 
^*An examiner should fiever cut 
short the time specified fpr a * 
test even though nil pupils hav€\ 
finished/' T or F - ^ 
which was answered correctly by 
only 60 per cent of the examine^ & ^ 
The pertinent statement in the' 
manual readSj "If all pupils^ or alj^; , 
but two or three in a class flnisp v 
before the stipulated tlrt^ has 
elapsedj time may be called.^ 

It is impossible to suppose 
the probleni here is one of read^ J ; 
ability or comprehensibility of flie 
material as presented* One must. ) 
seek the explanation in the atti- 
ttjide, or set, or motivation, which 
I uscTs or in thiicase the exam^ • 
inee, brings*" td" theftask of reading 
■^e m^^uah It is n^rd to. believe' 
*tibat the average tea^^r, if suffl-^ 
.^^fently interest^, cOuld^^t lo^te 
in a test manual information ot 
the kind called for in the question 
just cited. Perhaps our real prob- 
lem is that of , dfeeovering .why- 
Ihef are not interested and how 
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Ais Inteeit can be itimulafed^ — ^ 
whidi fi^gi'us bade to Ae prin- 
/ eiple aaipey statedp thai testing 
and t^t results take on meanfag/ 
when the user |e^ tiiem Iri rela- t 
tion t p his own needs, and prob-' , 



ii unquestibiia^ an urgent^ 
need' for better ^mmunicatiqri 
fro^ test maker to test user, if list 
result! aie to be made more mean- 
ingfuf to the ayerage user. 
T^e test manual can b^ mdfc a 



Infonmatiori of ^e^kind Revealed in. 
diis study t dis^ncertingj if igt eit 
tiraly un^ect^l but it is n^ssar^ 
fdr any realistip appraisal of wnat t^t 
manuals can r^SMiablV b^^q^eted to 
do, mnd as a ba& for thAf impr^ve- 
me^* I wi ^d to note, in ^is con- 
h^on, tMt Rbger^llispri of Educa- 
tignal Testing Set|?ice' has been makings 
a^emewha^siimijAktudf of the useful-, 
ness of die manual l6r the ACE Psycho- 
k^cal fibminatidn. Hi h%s prepaj^d 
tWo fornfir of a test intended|to reveal 
tiie extent to^hich'' teachers of^ equn- 
selors caiv^HlSice effecti use of 
r^ults^ on the basis '^inf^feation. wP 
sented in its manuah ^fe^s, moreoiprj 
^Ueited their opinions as to^ the ade- , 
q\iaoy of Various sections of tlio manual, 
.fotth^lnfomatiOT they wpuM Hk^ to 
hA^eaneluded, etc. His flnAnpras fa^j 
afif r know, have not yet^een madt^ 

public* t ^ ' 

*To summarize-, we have considered 
the test manual as a me.^nj of transmit- 
ti^ to the us the informatfon and 
skiljs he needs* If the refults a^ to be 
meaningful and useful to him. ,,Th6 fol- 
lowing* pointa^^e been advanced. 



1 
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more effectiw Wmffltinloatiori^e^ 
dium^ and can contribute more to 
l^ettef understaiidintf of test fe- 
suits* ' 
3^TWe hi^aningfulAess of test results 
t§^ the average uier will be en- 
♦V lanced if the manuaV concerns 
itself^ to a greater extent with re- 
lating test results to the user's 
problems and needss and demon- 
strates specifically and concretely 
the kindi of actions in^ated by 
the results, 
4. The readability of test manuals, 
' and presumably their eomprehe^- 
sibility, An be improved; , but it 
<i is neither possible nor desirable to ; 

avoid technical nfateriah 
^5. Depci^len0e^n the test tnanual 
alone fg^pjoper u^derstaridtfig 
- and^i^Sf test results is inade- 
quate. Not only must more formal 
training in measurement be en- 
courag^ as vitd for proper use of 
' test results, bur such formal work 
should ho- supplemented by work- 
shops and other in-service Gain- 
ing, , if teachers, counselors, and 
si^^Tvisors are trf derive -maxi- 
mal benefit from te^ts.^ 
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The Ta^ching of'EducatioriaJ Measurement 
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Jm INfRODUCTORV COURSE IN 
DUCATIONAL' MEASURE^JENT* 



A EECEPTT iimvEY^'ofrpiyahologisti em^ 
ployed in' sdiooi^^ jepai^enta of 
^ucatidn in 339 instituUoni provides 
among bttier things^ a Bst ot the Qoursea 
which Aey teach together with the 
frequency of mention. Among these are , 
many courses whose title includes 'the 
wgjpds measurement *or testing. In fflctj 
kfluch courses cqnstitute^ one qi the first 
Jten in frequency of mention in"^ list 
^di 43. Ite .most coupon titles of 
^ measurement courses are Tests and 
Measufem^tSs^,^ducationBl Tests kjid 
^ Measureinenti^ Mental Testing, w ad- 
dition dierfe-if^a siArtantial number lof 
measurementf %oiii^ Wliose titles ar^ 
not readily classified under dies^e three, 
t Since these courieSi however namedj 
are aU offer^ injschools or departments 
of education iti may be taken lor 
granted tilat oAOat of therti, if not all, 
are ' designed and offered for prospec= 
. tivf^ teacherSi counselors and s Apfel 
psychologists of mose already in such 
positions* It seems likely also, that if 
'^i schools or depa)fttnents of education 
Naffer any bourse in testing ^offly ofie 

* such course it probably w$uia b&an 
troductory coufta in v educational meas- 
urraeiH* In order to check thi^ line of 

^1 reasoning the catalogs of four types of 
y ihsti^tions Af ere examined, TTiese four 
were (f) larja pubM^KPorted col- 
V leges and universities, Ifrj large pri- 

* vatmy supported Ins^IfiiSSnsr (3) state 

* teadiCTS collegesj and (I) Better loiown 

J 



li*beral arts colleges, /The ny^ber of 
leach type of institution was approxl- 

, mutely the ^ame. All toldj' tfie catalogs 
of 6B institutions were studied, * . ; 

Although the analysis is maomplete 
certain interesting facts are ffflrly clear: " 

, Firsts it appears thst an ii^oductoty 
course in educational measurement is 
offered in nearly e\ ery institution of 

' the first three grou^sp but of the Ubefal 
arts colleggs only about ont half tot it 
among their offerings. The coia^vis 
usually either undergra4uata, or open- 
to both undergraduates and graduates; 
in'fqur institutions the cqurse i^ffered 
in the department of psychQlo^; ini 
all'othe^ |t Is offered in the departmfenfr' 
or schorf of^education* The usual nuu^ ^ 
' ber of credits is 2 or 3 semester hours.; 
It is gerierally an elective coursp though 
16 institutions require it of under^adu- 

**ates irt' ' teacher edncation « and about 

, hQlf th^many (7) say it is r^^ired of 
gradua^te students in education, Dat% ' 
on the^e matters as ^eir'as on pre- " 
re^uisiteSi purposes of ^tha Q||iirse, re^ 
^ quuenaeht for a teacher'Olicense and 
Qtlfct considerations are b^ng collected* 
It sterns evident th^t tlfe inttoduc- 
tory course in educational Measurement 
is a Ai^atter of imfiortance in pro-ams 
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for the f ducation of teadieri and ^ 
. tiiVs©; concerned with measurement' 
powses ^ a field of specialization, 

Furposes of to tourse^ 



of tfiis first course in edycational ineae 
urement? Naturally, ffiese wUl vary" 
somewhat from sitaation to situation, 
bflfrtiie foUowin^eam important and 
prapticaL ^ ^ 

tft) Qfi^nlaffon^^T^ student hm 
undoubtedly heard - pf various .types of 
t^ti such as tfchievementi intelligencej 
personalityi etc. -He has taken some 
constructed by t his teachers and per-_ 
haps sdrne standardized tests. He prob- 
ably has heard tfemis like validity^ 
norm^, standardized and pe^rmance' 
,but ^is/is his inttoduction To a sys- 
tematic Lptelantation and. definition of 
such concepts. In one sense, the. func- 
tion of orientation ma^f be the Imost 

■ important for thisYcOurse. Certaii|ly it 
is contributed to by practieally Every- 
thing "that we do in it since so much 

' ©f it is new to the student, However, 
orientation is a very broad term with 
many possible facets. Itvis necWsary to 
choose the lines aloni which it can 
most profitably be dirpcted sinbe we 
should not attempt to do too much in 
this course, which for many is their 
only one in measurement^ Nevertheless, 
.it seems that there should be.orienta- 



tion with respecT'*t& the following, at ^ 
least: * ^ k 

Uilderstanding of basic concepts of 
measurement such as, validity, celia- 
bilit>^ and other criteria of good meas- 
uring instruments;^ different kinds of 
V tests and other evaluative methods and 
devices; statistical concepts and some 
practice in simple statistical techniqjLiTM| 
some acquaintance with typical ^tw^ 
ardized t^sts of the principal kinds;- 
acquaintaiice with several standardized . 
tests in ^e students' own major field; " 
and knowledg| of sources of standar^- 
^ ized tests a^ Information about the: 



( b ) Improvemmt tmcher-mad^ 
, tests — Every teaser must test and 
evali^ate, if hgthin^ more than at lerfst 
'the ^status ana progress of the learnW* 
in subject-matter. ^ ^would appepr, 
tliergf or e Jhat an to^jtant function of 
,^he introductory i course in.'educaMonal ' 
measurement should be the improve- ^ 
mentV practices of teachers, that is,^ to 
Jielp them^ao better what is an iiil^al ^ 
\part of the work 6f a teaahcr. . ^ 
V (q) IhtroducHon to tlk im of stand- 
ardized tests in the school— This in- V 
^u^^ ^owledge of soipces of tests 
land q|itical tiiformation/ about them; ^ 
She aoilfty to read test manuals and ^ 
mak J judgmeBts between itandardiied 
tests in the i|ht of the conditions ex- 
feing k\ tllS"school where theyfare to 
be given and of objective data about '* 
them; how to administer, score and 
analyze results of standardized tests; 
and how to use this knowledge in "plan- 
ning and carrying out a practical test- 
ing pro-am which contalButes *to the 
solution of cducationril ^rdblems anU 
^furthers the purposes oJ,the sghoKQl 

To reta^pitulate, th^ n>aJor purposes 
. of thg first course in e&eational pteas-^ 
airem^t are conceived to b^. (a) oHen- 
iation to appropriate elemenfaty phases ■ 
o^ihe subject, (b) impfovement of 
Reamer- or locallij-mflde kests (c) in- 
trodiiction to the two of standardizejd 
Jests in4he school. _- 



f 



JProeedur^s ■ ^ 

* A^ether one accepts the purposes as 
stated or holds to a differerit set j0l 
goals* the logical question in qjiy caie 
becpmcs that of how to aihie>{& them.^ 
The remainder of this^aner fe devoted 
to discussion of methods lor doing ihis. 
No clairfi is made tha^4the ideaT.are 

/ rtiif^fntiAinm^l SUCCeWuh 



unique, or ' outstandih^f i sue 
ner^b 

number of years of txperi^ce iri^teach-^^ 
se(ar 



Thoy represent mer^l>r the results pf a 
pxper 

ing such a course tand an interest in 



cxpcrimenyng with various methods, in 
the hope ot improvi9%'it.^ 



/ 



(a) Orientation^ 

^ During flie orientation phase tiie 
main' aafcivitiei are lecture, class dis^ 
i<cujsion and problenii.^The ycturea and 
class diseusiions' cmter affpund basic 
concepts^ differant londs of itandard- 
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instruments, and statisU^.' Specimen 
sets of standardized tests are brought to 

, class, itndied and discussed, ftebjems 
affe assigned in making" a frecfuency 
table, calculating me^ins and medians, 
semi-interquartile range and standarfi 
deviaUon, rank difference cozrelationp 

»and setting up an ^ ogive curve. Per- 
centiles, standard scor^ and quotients 
are also iqbroduced and discussed at 
this -time. An at||nipt is made to bring 
out meanings ' and useS^ of ' statistical 
measures radier dian to emphasize cal^ 
oulation, but each member of the class 
works out the assigned problems which 
are graded and rehimed. There is no 
illusion that these ideas and techriJ^ues 
are mastered by all students or even the 
majority. However, an attempt is^made 
to give ihem enough understanding so 
that they can at least read reports and 
articles bas^d on resulte of testing with 
fair comprehension and to develop 
some understanding and appreciation of 
statistical metliods as an essential tool 
for deriving.meaning from raw scores or 
other test data* 

( b ) Improvement of Teacher^ or 
Locally*Mad^e Tests ' 

i^s was said earlier, every teacher 
must te^ and evaluate. It is ' felt that 
improvement i^ this phase of the work 
can best be brought abctut by actual 
practice in setting up objectives, mak- 
ing test itemt, and building a test. Im- 
mediately following the mid-term exam- 
inations mimeographed sheets .are 
handed out specifying in detail the na- 
ture of the assignment.' Briefly stat^, 
it includes the construction of a 100- 
item objective tfet,^ together with in- r 



' ^shiictions for giving and scoring 'it, and 
a scoring key, A statem^t of die objec- 
tives of the course in whjch thrf test is 
designed to be us^d and an indication 
of which objectives the test has been 
designed or is thought to measure are 
-^also-requiredi- 



The full time of the class is devoted 
to this projectglHr approximately three 
weeks. One of the first activities is 
examination of standardized te^fc in 
fields requested by the students, usually 
their major fields or subjects, and of 
tests made by students in previous 
classes in meeting this same assignment. 
' The class is divided into ^x)ups accord- 
ing to fields of specialization such as 
reaHing, science, homemaking, ^etc,. A 
large room, preferably wiih tables and 
chairt, is best Jor this purpose, The 
instructor brings to class specimen sets 
ofulifFercnt selected standardized tests 
in^ach of the desired fields and sub- 
jects. The various subject matter or 
interest groups then work over tliem. 
Next, student%made .tests are brought 
in and^used in tl^ same way. Three or 
four class meetiilgs 'Are usually enough 
for this part of the project. Students 
who want more time for it may use tlie 
departmentdl test files during office 
hoiurs. In general, the students avsf not 
encouraged to take these teste out of 
the classroom or office but they may be 
permitted to do so by individual re- 
quest. 

, While this is going on, students are 
given spme help in thinking about ob- 
jectives. Suggestions are made on dif- 
ferent ways of stating Uiem and the 
desirability of stating tliem in terms 
of pupil behavior rath^ than subject 
matter is -stressed. They also examine 
the manuals of both standardized and 
student-made tests for such statements. 
Visits to the so-called Juvenile Collec- 
tion of elementary atid secondary school 
textbooks in the library for'ideas as to 
objectives and content are encouraged. 
Stu Jents are strongly urged to plan^nd 
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consbiict a test at tha level at which 
they expect to teach. This ii epnsidered 
desirable because the students' subject- 
matter competence is usually *inucji 
bett^ suited to fliis^than to a college 
or diversity level test Equally impor- 



be, much tippreciated by the students. 

Both the Jriginal and the carbon 
copy ard turned in to the instructor* 
The original is^carefully ' ^iticlzed,, 
evaluated, and returned ^wjth' a mark 
to the studcnt^author. The carbon, un- 



or ^verslty level test equally impor^ vu -^^^ ri f^' tu^ n^lW^^ to 

H^ir-y^^onsfdlratfon^ 



makes a test for eWmentary or high 
school level he has son^tl^g^he may 
find quite useful in the not-too-distant 
future. 

* If students wish to attelnpt consfruc- 
tion of something else such as a rating 
scale or readiness test they are en- 
couraged to do so provided they and 
the ins&uctor a^e satisfied that they 
have the necessary knowledge and ex- 
perience to make die efer t worthwhile.^ 
Following these actlffes, consfdera- 



be examined for ideas and for cntimsin 
>y classes in succeeding .terms, 'When 
'the papers are returned the hour for 
that day is generally used to discuss 
common or outstanding strengths and' 
shortcomings. Students are also invited 
to ask for individual ^conferen^s wfth 
the graduate assistant or the instructor 
on the details of th^ project and there 
ar5 always some who do so* ~ 

Ovef tiie ^a^s in whichHthis assign- 
ment has been used we have come^to 



roiloWinE tnese accrues J uuAiait*ciii- ^^^-^ _^ 

Won is given to each of the commonly , feel that it is one of the most rewa^.ng 
• * . ^ ^ * . . :» ^^ju - r^^HifiHsc fif Hifi f^iirsR^ Students occa- 



used types of objective test itemsj Each 
type is illustrated, and its advantages, 
disadvantages and particular useful- 
nesses are briefly discussed, Students 
are invited to try making sample items 
which, diey may hand in for criticism 
and suggestiona from the insfruetor. 

Ele'hi^tary principles as to format 
of a test, grQuping jhd arrangement of 
items, setting up of scoring key or 
stencits, directions for taking the test, 
for administering and scoring it, and 
i use of answer sheets are also taken up. 
After these activitfts in class ar#con= 
eluded students generally have about a 
^ week or ten days longer to 'complete 
the project and hand it in, usually a 
vi^eek before the end of the term. They 
are urged to type the entire paper and 
make a carbon copy. Thcfe are two 
reasons for thj|. First, if the test is 
typed it is rtftc}i si^^et to control 
spacing and ^lignptfnt. In additibn, the 
^student-author u|ually^;has ntnre pride 
and satisfaction^in a neatly typed paper 
than one written in Idnghand. The soc= 
ond reason was alluded to earlier, in 
that thi carbon copies are filed- and 
used tm instructional purposes with 
succeedirtg dasses. Tills has proved to 



activities of the coursei Students occa 
siooally complain that it takes too much 
time and effort but the much more com- 
mon reaction is one of— ^"It took a lot 
of vvoA but 'we learned a ''lot and it 
was worth the trouble." It. often opens 
up an entirely new concept of measure- 
ment to the ^udent. It gives him^soma 
^now-how in forniidating objectives of 
nstruction and trying' to devise test 
terns to measure these objec^ves. It 
4^iVes him an appreciation of the care 
and effort requirtd to consthict a good, 
objective test. Finally, it ^Ives him a 
sense of prido in authorship and in 
liaving produced something tangible 
and substantial which he takef with 
him. and possibly finds use for, long 
after most of 'what we hy to teach in 
fho course may be forgotten. 

Although ^s believed that thp bene-' 
fits" to the itudent of this exercise far, 
Gxcekl its disadvantages we recognize 
that it lias scuttle of the latter; For one 
thing it is tjtnc-eonsjttjning. ?A good 
share f*f tlic time of tire last month of 
the course, both in class and^out of it, 
is giV^h over to the project. Some may 
question whether , the practice in test 
planning and construction: merits that 
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much time out of % term. We deflnitely 
feel tfiat it does for reasons akeady 
stated. ^ 

Anothfr possible ^itieism is that the 
student mayi in ' his inexperienqe and 
luck of interest, be impellfid to lift items 

^iHit leart, ident fat ifemHrfronrfouloeg ^ 
ayailable to him andj^ot^do ^l^c^l origi- 
nal work or thinking at^ll. Ever^ effort 

; is , made to prevent this.' Eyery bppor- 
timity is taken to emphasize the.praati' 
eal value of the exercise and to point 
out tjjat stiidents can obtain the maxi- . 
miim benent from it dnl)i by goin^ 
tlirough the. entire experiehoe for them- 
selves. Also, they are told that it is. not 
difficult for the instructor fo distinguish 

^between items' taken from existing tests 
and tliosa. original with the student. 
T^re iSj of course, an element of 
bluffing in this, but it has been possible 
in some eases, from time to, time, to 
rfetect such cheating and Imve the stu- 
dent confronted with the evidence^ own 
up to it, In the great majority of cases, 
our impression i^^at they do not 
knowingly borrjw from other sources 
for test itemsT^ ^ 

Anotlier disadvantage is the gre*1t . 
amount of time required to read the 
papers. It takes on the average, one 
half hour per paper to read, criticise 
and evaluate. The procedure fbllowed 
is to hdve a graduate assistant sit in th« " 
course, especially during the second 
half, so that he knows the assigument^ 
and emphases thoroughly and also gains 
some familiarity with %e apafe hi which 
students are working. By the time the 
phpers come in he has been briefed on 
procedures in evaluating tb^, \vhat to 
look for, etc. He reads each one iare- 
fully, mak§s notes, and gives a tenta- 
tive evaluation. Following this the in- 
structor looks them over, adds or 
modifies comments and, if he disagrcDS 
with the .assistant's evaluation the paper 
is di^ussed with him and a decfsiohNs 
rpaetied as to the final mfirk. All this 
takes time and energy but it is felt that 



something into which students have put 
so much^^time and effort deserve 
carefiil, *eonsci^tious^ reading and 
bDneflt of a.ny Mig^estions we can pffer. 

One of tHfe intof eating sidelights on 
this prefect is Ae vafioty of t^sts tliat 

~jTav3 accilmulared in ^ flie ifisfructor*s 
files. ovii? the^years.*Be^des all the do^- 
nionly taught brancri|s in whicli stand- 
ardized'''teste are qvailable ^e list in- 
cludes tests in e^^ery major sport; the 
various phases of, industriaL arts atid 
agriculture; speech, TadiOj Und journal- 

^^i; corrimercial subjects! and tes^* on 
two Bibles^the King James version 
and the Freshman Handbook, 

i ( c ) Introduction tp the Use of 
Standardised Tests In the Sehobl 

in planning a testing program for a 
school in such a course ad this it is not 
uncommon .to set up a ■hypothetical 
situation as a basis for the discussion. 
The instructor 'supplies such data as 
size of enrollment, number of. grades 
and pupils' per grade, data* available 
from tests previously given, if any, 
amount of money to be. silent, and simi- 
lar pertinent matters, Beguirring with 
these data a testing program for the 
fifst year and for succeeding years may 
be planncdi Practical considerations 
such as selecting tests in thejight of the 
purposes., to be served, ordering ^the 
test^,, ^curing cooperation of the staflj, 
administering and scoring the tests, and 
analyzing and using tHe results arp 
*among the points usually considered. 
While this exercise is not* like^ the aclual 
experience it may provide some iiisight 
into th^ rcguiremehts ajid tlyfproblems 
of an effective use of tests in the 
schools. Nevertheless, it c^not take 
the place of actual participation in plan*v 
^ing and parrying out sucli activities. 

It is quite often pc^sible to %tve stu^' 
dents actual expedience and participa- 
tion in a testing project or program. 
Requests are often received . from 
schools for advice tod aSsistanc^an se- 
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lecliVjg' tests). planning for their .admin.' 
istratiSn, administering and scormg 
them, analyzing the results, and study- 
ing them for ways ki which they can 
be used. If it can be an-inged without 
> too much inconvenience or loss of time 
' we try to give students taking nreas^re^ 
ment coui^ses an. opportunity to,^artici> 
pate in such activities in a^way that^^ 
be beneficjal to them and to thd sqhotel 
seeking help. Sludents in the mtroduc- 
tory^ course in educational ineasurement 
as well as . those ia later courses in in- 
dividual exainining and group, testmg 
% hav^ frequently takeii part in ^various 
phases .of testing pragrams with which 
we h^ve-been involved, A more dcv 
tailed report on such participation flian 
time permits today has been given ebe^ ' 

where.^ ' 

Perhaps a school has decided to give 
a readiness test to first^graders; or in- 
telligence tests in aU elementary grades; 
or reading tests in secondary grades. 
' Whatever the nature of the tests to bo 
given, the piupose for using thein 
usyally arises out of a problem or need 
somewhere in the school situation 
which is described and explained to ttie 
class in measurement. 

Generally, the teat or tests to be 
given have been selected by tlic prmci-_ 
pal of the school or by a committee ,of 
teachers. We-ask f6r enough copies ot 
the tests so Uiat ' every member of the 
class may have 6ne. We also collect all 
the manuals and other accessories that 
^an be spared. Then the class is organ^ 
ized into teams of 2 to 5 members each, 
depending on the size of the class on 
the one hand and die number and size 
of pupil groups to be tested on the 
other, th6 tests to be administered, and 
' similar considerations. Iii the grades wo 
generally: work with the classroom as a 
unit while in secondary schools a wholu 
class may be tested depending on its 
size and the facilities available. Occa- 
sionally, if the measurement class is a 
large one the pupils arc divided into 



small groups so as to give biore students 
expertcnce; 4 team .k assigned tq a 
group and is cesponsible for testing that 
group. The team elects one of its num- 
ber as the cxai|uner and the rest serve 
as assistants or proctors. Copies of the 
ftst and manual are studied and duties 
of eacH team member are deterVnined. 
The examiner practices his part such as 
reading directions, timing, ' etc, and 
pVoctors familiarize themselves with the 
procedures and with their responsibili- 
ti6s. Mi^h of rfiis goes on outside the 
classroom but at least one class: s^sion 
is usually devoted to anssvering 4ues- 
Hons, and 4i general briefing by the in- 

structor. ^ ' 

During diis time. of , Reparation the 
dctailedr arranRcments^ve been made 
for the administration of the tests. The 
date and time are set^ermission is ob- 
tained for participating students, to 
make up work of classes that wdl be 
missed and, if necessary, transportation 
by college bus or other means m bx- 
ranged. It has been found preferable, 
if the class is large, to furnish transport 
tution to insure that all participants wiU 
arrive at the appointed time and place. 

It is always suggested to the princi- 
pal, as diplomatically as possible, that 
the regular teachers be given a free pe- 
riod while the testing is going.on. It has 
been found best, generally speaking, 
not to have the teachers in the room 
during the testing. The children ar^ 
usually thrilled at being viriitedjjy col= 
lege students, some of whom may be 
vvell-known for athletic prowess or 
other achievements, and there is seldom 
a prohlcm of lack of cooperation. Tho 
instrnctor has assigned teams in ad- 
vance, setis that Uiey get to their sta- 
tions, and stays on the job until the 
testing is completed. 

After the tests have been given they 

^ » Nt)ll Victor IL and Marvin 1> Glock, 
Functional Courses in_ Mousuryimjnt ami 
Evaluation. School and bocicty, 70 1 jaa= 
340. November 26, 1949. 
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are tskm by the teams and scored. 
Paperi are exdiang^ between teams 
and die scoring is checked. The results 
are analyzed by use of simple statistical 
procedure. Early in die course some 
problems have been worked out by the 
membOT of the class in which, such 
procedures practiced, Students 

who have colledted data by administer- 
ing tests to real children and scoring 
these tests, bjb much more sfrongly 
motivated toward siich work than diey 
were previously., With the help of 'the 
instructor jdie class studies the results, 
attempts to intei^ret .them, and per- 
haps formulates some tentative conqlu- 
sions and recommpndationSi 'If it can 
be ^gnged, the principal and perhaps 
a tocher or two meet widi the class to 
dftcuss the results.. "Oiis is usually a 
very interesting session. Studentt gain 
Wights into the situation which they 
wo^d otherwise nOt get and repre= 
sentatives of the school receive assist- 
ance in interpreting the results of the 
tests. In addition, of course, ^^ey have 
ihad the substantial Help of having the 
tests administered and scored for them 
and of having a preliminary statistical 
analysis ''completed. 

Students always seem to enjoy this 
proj^t and feel that it is practicaL and 
valuable. Most students like to give 
testk and they consider this an oppor- 
tunity to gain some aptual experience. 
Scoring presents no problems when the 
work is divided among thirty or more 
students. Sometimes they have de- 
veloped a personal interest in some of 
the children tested and are most anx- 
ioui to ^e ho^certain ones came out 
on the test. Likewise, analysis and 
study of the results to see what use can 
bo made of them by the school presents 
a challenge to which they react quite 
favorably* One of the objectives which 
students in measurement courses con- 
sider most important is that of learning 
to interpret and to apply results of 
testing. 
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From the schoors standpoint the as- 
sistance received is always favorably 
regarded. The average teacher in serv- 
ice often knows little about the use of 
standar^zed tests. When the project is 
carried out cooperatively as htere de- 
scribed teachers learn along with stu- 
dents. Tliey also may change from a 
negative or skeptical attitude toward 
tests td'a more redlistic one which rec^ 
ognizes the beneflts that testing may 
provide and not just its Um^ations. 

Because of its value to the greatest 
number, the test consmiction project 
previously described is always a part 
of the first course in educational meas- 
urement. The class participation in 
testing in the schools is included when- 
ever it is feasible to do so. The major 
considerations are time available, 
whether it can be fitted into the sched- 
ule, size of the class, and above all, 
whether the experience is such that it 
will be educationally valuable. We do 
not enter into such arrangements merely 
or primarily for the assistance and bene- 
fit of the school requesting it. Some- 
times the tests to be given or the grade 
level at which they are to be given do 
not appear to be such that the experi- 
ence seems very appropriate or valuable 
for a particuFar class. Again^ the class 
may be too large to handle easily, al- 
though as many as 40 have been used. 
Also, the mauirity' of the students and 
dieir readiness to undertake the project 
may be a determining factor. And,' of 
course, if it appears diat there is not 
enough time left in the term to do a 
good job, the "blass is not brought into 
it. 

It ftigy l|& appropriate to speak 
briefly doncerning the general require- 
ments of the course lest some listeners 
gain the impression from what has been 
said here today that it consists solely of 
the projects or activities which have 
been described. This would be a rather 
incomplete picture.^puring the orienta- 
tion phase, which occupies the first 
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mDhth or SO of tfie tenn, major empha- 
sis is placed on assigned readinp in the 
textbook covering basic Aeory and sta- 
tiitical methodsJ We have not generally 
required much outside^ reading in un- 
der^aduate lections because vve feel 
that reasonable attainment tiirou^ the 
*itudy of one basic textbook is perhaps 
all that can be expected of 'beginners- 
Students who evince interesy in par- 
ticular phases of Ae work afe given a 
supplementary reading list and assisted 
Jn finding additional material. 
^ - 'To* measure achievement we use 
three objective examinations in addi- 
tion to die problems and papers de- 
scribed earlier. The first of these deals 
entirely With elementary statistical con- 
cepts; die second is the mid-term ex- 
amination on basie theory and statisd- 
cal methods; the final examination 
covers the reading and lectures of th0 
entire coiuse. 

In closing, I should like to make ex- 
plicit two ideas that have been implied 
in much that has been said here today. 
The first is that a great deal of what we 
try to teaoh in introductory courses in 
measurement probably Ijas little signifi- 
cance to the* student unless he has op- 
portunity to make use of the ideas we 
present. Measurement theory, statiitics, 
use and interpretation of - standardized 
tests and test construction can mean 
httle to the person vvjio has not actually 
applied them or worked at them, This 
is not to say diat verbalization of term4^| 
Und concepts^ and classroom discussions ,: , 



and practide to educational measure- 
ment have no value; however, it seems 
reasonable to assume that apphcayon of 
these ideas .and procedures in ^^Sore 
realistic situation would increase both 
motivation and understanding. The 
value of our courses in educational 
measurement probably depends for 
most students on the extent to which 
what is taught carries over into practicei 
If this is so, every effort should be made 
by instructors in such courses to facili- 
tate and increase this transfer, 

The second point, which is closely 
related^to the first, is that it is possible 
to carry out activities, even in begin- 
ning courses in educational measure- 
ment, that will provide j^ptical oppor- 
tunities for students to apply what is 
presented in the course, The projects 
here described provide experience and^ 
practice in the application of. measure- 
ment theory and techniques to situa- 
tions which are like those many 
prospective teachers will face after 
graduation. Through activities of the 
types discussed, we aim to give stu- 
dents somethJng useful and basically 
sound in the field of educational meas- 
urement. No claim is made that what 
we do is very original or unique. What 
has been described here today is of- 
fered merely as an account of some 
procedures that have been used suc- 
cessfullyi with the hope tl^at it may be 
interesting and possibly helpful to 
others responsible for the same type of 
course. 
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IN^SEftVICET^RAINING IN MEASUREMENT BY MEANS OF' 
UNIVERSITY EXTENSION COURSES 



A^rNUALLY Harvard-Boston Univer- 
sity Extension Division conducts flfty 
to sixty courses for teachers throughout 
the New England areaT During t^e 
1852-lQS2^^hool year, sixty-one differ- 
ent coursesvvfere offerid in forty-seven 
eommuniti^. Over the thirty-one year 
period during which the Extenskfti Du 
vision has been functioningj a total of 
794 courses has been pven in 159 
communities enrolling 21j998 teachers.^ 
If these adaptive training progranis are 
effectively organized, they should re- 
sult in improved school practices on a 
wide scmle.^ . ' 

Several courses have been called for 
with high frequency in the area of edu- 
cational measurement and evaluation. 
During the past nine years, the writer 
has personally conducted fourteen 
measurement workshops in the follow^ 
ing New England communities r Port- 
land, Maine; Barre, Vermont; Man- 
chesjter^ Peterboro and Hampton in 
New Hampshire; Framingham, Mil- 
^ord, Maiden, Norwood^ Stoughton, 
Rockland j Fall River, New Bedford, 
and Quincy in Massachusetts. 

All the courses are set up upon re- 
quest of die local school community. 
The courses generally carry two points 
of credit^ aldiough a diird point can be 
^ earned by special arrangement with the 
instructof* Most courses carry graduate 
credit and run for fifteen weekly meet- 
ings of two hours each, ^e instructor 



C9mmutes to the communityj which is 
generally within two or three hours' 
riding distance from the Boston Uni- 
versity campus. 

Other than for tljs obvious saving 
and convenience in^transportation for 
the class members, where do the ad- 
vantages lie in offering an off-campus 
course? In what way* does the off- 
campus measurement workshop course 
differ from* the course with the s^me 
title offered within the University? The 
majpr strategy lies in the following^ (1) 
the course is offfered only on community 
invitation or dema'ndj Oius indicating^ 
that teachers and school ^Kcials are 
bothered by measurement problems 
and are^ perhaps, in a state of readiness 
for learning j (2) the teachers are in 
close contact with rdal measurement 
problems around which learning experi- 
ences may be planned; and (3) course 
evaluations are now possible in terins of 
actual changes and improvements in 
the classroom behavior of the teacher* 
In view of these circumstances, the off- 
campus course in measurement and 
evaluation can be expected more read- 
ily to affect actual school practices* 

- W, Linwood Chase^ * 'Field Service," 
Boston Unimrsity Graamta Journal^ Vol. 
I, No, 10 (March, 1953), p^. 151'1S2. 

Donald D. Durrell, "They Are Popular 
With, Teachere," The Journal of the Na- 
tional Education Association^ Vol. 35 (De- 
cember, 1940), pp. 572-573. 
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Disadvantages in off-campus courses 
center ^around the lack of adequate li- 
brary resources. However, except lor 
bound periodical references, this has 
been largely overcome tiirough the 
estabhshment of a separate extension 
course library. 

Based on the experiences of the past 
nine years during which ^e writer has 
offered fourteen workshop courses in 
as many New England communities, ' 
the following generalizations have been 
drawn concerning methods and pro- 
cedures which condition the effective- 
ness, of an off-cainpus course in improv- 
ing meastirement practices in the local 
schools and communities. 

\\^e all measurement courses have 
been offered only upon invitati^ of the 
local community, much depends upon 
who Lnitiates the request or who spon= 
sors the ^urse. Courses conduced be- 
cause a supervisor or superintendent 
feels that his teachers are in dire need 
of tiii3 type of in-service training may 
result In an "increment-happy" captive 
audience lured on by monetary stimu- 
lus rather than by any desire foy self- 
improvement. Only as the teachers 
have been consulted and have them_^ 
selves elected to sponsor such an ex- 
tension cours^ will there develop a 
wholesome Iqahiing situation. Frc- 
' quentjy the local teachers' associatiou 
or a professional improvement or m- 
service training coniniittee initiates the 
invitation and sponsors the course, thus 
representing a true consumer dcmund. 
Under such auspices there is more 
promise for ultimate improvement of 
school practices. ^ 
The Harvard^Boston University Ex- 
' ■ tension Division has lent considerable 
assistance to a number of communitr-s 
in ,the sequential planning of courst^s 
requested over the years. For example, 
the measurement course has frequently 
served as a starting base from which 
have stemmed courses iti guidance, cur- 
riculum planning, etc. II an off-can^us 



course does not fit within the mosaic 
of the over-all in-service training pro- 
granV in the community, it may repre- 
sent a sponidic. ancF spontaneous flight 
in adaptive training, Suclv isolated 
course offerings eventually prove ^f 
limited vaiue. 

Prior to the ikst meeting with the 
local workshop group, the writer has 
followed the practice of spending at 
least a day in the scliools of the com- 
munity in order to get the "feel" of the 
present status of toting, with particular 
reference to the^ existence of cumulalive 
records, testing programs already in 
progress on an^ all-community basis, 
local leadership responsibilities for test- 
ing, curriculum revision programs un- 
derway, *natuve of. the present fonris 
for rcpovting pupiP progress, and atti- 
tudes (ft local principals and sup^- 
visors toward cunent measurement and 
evaluation practices. For example, prior 
to offWing a course currently being 
conducted m Quincy, Massachusetts, 
an afternoon was spent in the schools 
of the community. A conference witli 
the local superintendent, assistant su- 
perintendent, guidance director and 
elemeiitary school consultant was held. 
V Tlus conference bTOught out the follow- 
iilfi types of informationr much thought 
was being given to problems of jeport 
cards and other forms for reporting pu- 
pil progress; thc'Stanford Achievement 
Test, Pavtial Battery, Fomi J, was being 
administered to all children in pades 
four through nine; and that tests of 
ability were being administered in 
grades one' and five. This^type of in- 
fovmation. enabled more effective plan- 
ning for meaningful learning experi- 
unees related to 'the testing program 
already in progress. 

While most of the .workshops have 
been eonductcd on a one-term basis 
and run for approximately a fifteen- 
week period, a promising adaptation 
has been made in several communities 
by extending the coiu-se throughout the 
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, year. In this way, the course meets 
every iliree or four weeks for the same 

- number of fifteen meetings^ thus ena- 
'bling projects to be planned and car- 
ried out, A measurement course con- 
fined to fast sequence of fifteen 
meetings seldom enables the 'teachers 
to do more th^n plao for the cons^c- 
tion and use of tests. A course spread 
out over the year insures the oppor- 
tunity to administerj analyze, and use 
the various instruments and techniques 
discussed Jn^the course. 

? 'In one communityj Framinghami 
Massachusetts, such a^workshop was 

" planned on a full-year basis and in- 
cluded all elementary teachers. This 
course met or^e afternoon a month on 
' school time. Glasses were dismisped foE 
this Aftemoonj ^ enabling a three- hoitf 
work session. Gerilbar meetings were 
held the first half of the period, foh 
lowed by^snl^ll group moetings with 

. committees working on different^ prob- ^ 
lerns in the second half of the session, 
Mfjeting on school time gave added 
prestige to the importance of educa- 
tidnal tasks undertaken in. this in-serv- 
ice, training program. Special arrange- 
ments were made'^by the community to ^ 
bear, a share of the expenses of the 
course, since many teachers were en- 
rolled on a njpn-credit basis, v 

In the flrsf years of the writers ex- 

..perience in conducting these coursers 
in the field, a comprehensive inventory 

" test covering the measurement and 
evaluation area had been prepared and 

'was used at the first session of every 
course. These inventory tests revealed 
that teachers had little knowledge of 
the field of measurement and "that they 
labored under a great deal of misin- 
formation. One oriier efFcct was tiot- 
able: use of the inventory instrument 
always tended to cut down on the bn- 
Tollmen t in the class, Xhose^people with 
particularly low scores dropped out of 
the course in greatest numbers. Teach- 
ers most in need found the initial test- 
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ing a traumatic experience and dedided 
not to take the course. Since repeated 
use of Ihis ^insfainnent always seemed 
t6 result in these two pjienomena, the 
w^ter now assumes that the teachers 
Jby.and large enter the course as begin* 
ning learners. At the same time, it is 
acknowledged that the size of the en- 
rollments are larger. However, there is 
considepable merit in discovering those^ 
perspns who have had recent courses 
in measurement and who do have more 
knowledge and skill than others, At^ 
tempts are now made to uncover these 
people through non-test tecRhiques in 

* order 'tp capitalize on the proficiencies ^ 
which they bring to the class. 

iLis vital to discover and work with 
the mdividual *who has jbeen assigned 
responsibihty for overseeing the testings 
aety^ties jn the local school system.'^In 

^ one community this may be the super- 
intendent; in^ anothe^it may be the 
supervispr; and, in stul another, the 
principal of the school may make de- 
cisions related to test constructionj pur- 
chase and use. If the mfeasuremcnt ■ 

' workshop is to be effective, .this indi- 
vidual should be a participating ifiem-' ^ 
ber of the group. It is easy to ^ec^how 
a course conducted by an outsider may 
nm at cross-purposes with a testing 
program planned l^y another individual 
wdrkiug within the school system. 

It is significant to ndte that many 
comniuniti^ had ifot assigned this re- 
sponsibility to any one person and thatj 
by and large, no qualified school person 
was^ available to assume such over- all 
sthool responsibilities. During these 
workshop courses, this problem has al- 
ways been faced by attempting to lo- 
cate those teachers or administrators 
who had had some previous training 
and experience in the use of tests, and 
through the establishment of a Local 
Evaluation Committee consisting of 
representative members of the school 
staff. One outcome^ of the course ob- 
served in a number of communities^ as 
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in Maiden, Massachusetts^ was^ tfie 
establishment of a ppsition of Director- 
of Measurement and Research which 
was evjentually "filled *by a person- 
Gained for the task 

Whenever ™^ible, the attempt, was 
made to limit .the t-ourse to the teachers 
in one town or city in an effort to plan 
the workshop ground the i^ique prob- 
lems of a^pahicular school s^stenti. This 
has not ^ always been po^ible, since 
some courses have been se^ up initially 
on .a regional, basis and have drawn 
from many small to%vns that by: them- 
selves did not have sufficient Bhrolirtient 
to support a coiu-se. Curreptly, such a 
course ir being offered in Ilampton, 
New ftampshire, composed of t%vcnty- 
' three teach ern representing a half= 
dozen different nearby school systems. 

In each Gommunity the course con- 
tent tended to center around the same 
core found in any beginning measure- 
ment course. At the same time, each 
community's requirements have called 
for some Ibcal patterning and for 
varied emphases oh particular topics. 
However^ in every community ^ the 
learning experiences that were set up 
for the group varied considerably. For 
example j one *^comnhinity may place 
emphasis on local test construction, 
going tbnnigh the process of item anal- 
ysis and local norm-building^ another^ 
community may center its efforts on the 
administration, analysis and classroom 
use of standardized tests; still another 
may spend a large share of time in re- 
vising its cumulative records and pupil 
report forms; and still another com- 
munity may take off the failing pupils 
in their class lists and do a thorough 
, individual test^ job with each young- 
ster. No school system showed any 
shortage in testing tasks to provide a 
rich source of learning experiences. The 
particular advantage of an off-campus 
workshop comes in^ this rich oppor- 
tunity for providing real-Ufe classroom 



situations involving knowledge of * and 
skill in, testing. 

In a number* of ^communitie^i tl>e 
superintendent' of schools has ^ made 
available A testing budget which gave 
the' work_shop members an opportunity 
to plan and carry out a school testing 
program, Such administrative coopera- 
' tion tended to insure a desirable reahsm 
in the c6ur.^e, since the group did mpre 
than analyse tests that might sometimes 
be u^d in a theoretical school system. 

Ipl^aluating learning as a product 
of ftie" course, use has always been* 
made of a carefully constructedr item- 
, analyzeci' achievement test prepared by 
■the instructor, together with 'an ap- 
^-ipraisal of the individual rfnd group ac- 
^fwities carriecP ^n by class members. 
Ho%vever; this final examination has al- ^ 
'ways been drawn from'a pool of avriil- 
_^ able items in the instructor's files, which 
are selected to cover the areas of em- 
phasis made in the particular course. 
By evaluating the aictual use made of 
the tests constructed pr purchased and 
which have been used in the classroomj 
and through school and classroom visi- 
tations, .some subjective impressions of 
tlie successful implementation of testing 
practiecs have been obtained in all 
fourteen communities. However^ the 
problem of evaluation remains a serious 
one. The wt iter 'looks to the day when 
^, courses sucii as the^c^will be given on 
^ a non-credit basis which will, in a 
sensCi evade the necessky of marking. 
This assumes in advance a professional 
maturity on the part of the in-service 
teacher and adheres to the concept that 
the activities engaged in are oriented 
in the direction of need and are basi- 
cally desirable school activities. 

Generally^ a person working in the 
field disco;verj^ that testing is somewhat 



- W. Q kvaniCGUS, "Adapting a Univer- 
sity Extension Cour.sc to tlie Ncyds of the 
Loral qomniunity,*' Sahool ami Society, 
Vttlunie 68, No. 17^4 (August 7, 1948), 
pp. 81^84. . 
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in bad . odor. T^e teacber is ponvinced 
that iomeone wants to test her— not the 
pupils since this has frequently been 
her experience in the past. At tne same 
time, most teachers show "a* severe 
allergy to mattprs^ smtistieaL Ftrthefr 
mor^s local leadership in the area of 
ipeasurement on an administtative and 
supervisory level is either ineffective or 
lacking. Through the off-campus work- 

S 



shops, the measurement man has a riel 
opportunity to help local ajkninistra- 
tion face the problems of fidaptive 
trai^ng in measurement and evalua- 
tion. It is in this area that the critical 
requirements in measurement and'ap-- 
praisal are- such as to make the diff^r- 
4nce between what is an butstandi^^y 
effective or a deflrritfely unsatisfactO£y 
teaching performance. 
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G L I-K S E N 



m^^mG FOR RE^A to^N feyCHO^ 



BS'^ortE ^u^iNiNG the type of tTaining 
i wirici I^ould eonsiderippropriate fon 
'rresearQh psychologiGal measurement/ 
let us pcmside]: briefly die nature of the 
problems of ^^psychologicah^teure- 
ment, in .order to . indicate tiie type of 
goal toward whieh' tfiis gaining i| di^ 

rected, » ^ Yr 

If one smreys the differeHt problems 
which have ^engaged ^the attentipn oi 
psychologists, one finds that tiiere is a 
■ certain class of problems which are met 
wift repeatedly atid are charactCTisti- 
cally piychological The major feature 
of thesQ prdtlems ire that die psychp- 



logical objects being studi^ are 
cally qualitativf in nature. Howe^5. it 
seems realonaWe and valuable to de- 
scribe tham quantitatively^ 

Fpr ^ample, inteLUgi^e is an ii^ 
portant psycholcrgical_;.^St of individr 
uals. It is basically a"" qualitative trait, 
.yet it seems reasonable both to laymen 
and to psychobgists to speak of one 
person as being more qt less intelhgent 
than another, inlplying quantitative 
differences in intoUigence. Tne meas- 
4HWient of. intelligence and odier men^ 
tal tfaits, and the determination of thei? 
interrelatlohships, is a large field 6f 
pS)wholopeal investigation. It is impor- 
tant boOi for psychologieal theory, and 
Ifor practical applications of psychology. 
&Wespondinily, sensatiom are basi- 
cally qualita^tive. The differertees be- 
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tween differenfi odorfr^e^^een differ- 
ent tonesj or ^etweeii different eqlors 
are clearly 'qiSlitatixe/ differeyices; yet 
we have a two dihiensional representa- 
tion af tones,'^three-dimensional color 
pyramid, and^ attempts to represent 
odorp in^three-dimensional space. 

Attitu^w of persons-' toward various , 
social prfelemi, qr individual judg-^ 
taente df valye, are ^^dditional illustta-^^ 
tions of psychological objects which are 
basically qualitative. Yet th§ stud^ of 
problems of attitude and vdue is aided 
tremendpusly, when attitudes are quan^ 
tified and measured by various attltud^ 
scales. It is necessary to know l^ow 
measured values before laws regarduig 
value can be j experimentally investi- 
gated. ' ^ , 

There are a great many more illus- 
trations of the iiyportance of quantifica- 
tion in the approach to numerous 
psychological problems that initially ap- 
pear to involve a field that is essentially 
qualftative. Tliese problefhi are the 
basic problems of psyehomefricf. One 
method for recognizing such problems 
in common-sense conversation is to 
note thafa perton discussing the prob- 
lem is hkely to say, "It Is all a matter of 
OpinicMi and you cant even get agree^ 
ment among thjfe opinions of different 
persons." This statemerit. could readily 
be made about attitudes, values, sen- 
sory qualities, etc. Many of the methods 
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of Psychological Measurement were de- 
veloped pr^isely to deal with this 
. situation, where the scientist must deal 
^ ,with opinions or judgments and the^e 
^^pjnions disagree. ^ 
A sligfitly different way of viewing 



to each stimulus. Mentartest data also 
fit the samfe general format^ since the 
Sg may represent either an item, ori^a*' 
test. The Rig then represent the per- 
'"son^s c6rrect or incorrect answer to each 
itenis or represents his score on each of 



these problms is illustrated in Table 1. ^the test^ fhd fact that the basic data 
We may, say that many if not most psy= of test ^bory, and of psychdph^sics 
chological problems ar^ concerned with ^cap be represented as matrices, indi- 
some aspects of the responses of per- cates tliat tlifnkin^ precisely about 

Uiese problcn^' would probably be (a= 



, TABLE Jl. 
STIMULI 
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person 
i ^ 1 



may be 
.. N). 



sons to stimuli. Each 
de^gnqted Pj (^where, 
Each stynulus may be desigiia*fed^ 
(where^g^ I , . , K). Such a set of 
psychological data for the responses '^f 
N gerspns to K stimuli caq in all, or 
practically all cases be represented as a 
mab"Jx, with {say) a column for each 
stimulus j and a row for each person, 
|ie response of the i4h person to the 



c^tated by matrix algebm. Many of t^e 
developments in both psychophysics 
and test theory rfver the past ten years 
or so * demonstrate the usefulness of 
matrix algebra in developing -the. soiu- 
tion to various problems, 

If onn tf^ards the stimulii the tests 
Or the iterif^ for example, as standard 
and attempts to^measure inter-individ- 
ual differences we have the problems 
of mental tests. If one assumes that the 
individuals □institute a some^^at ^and- ' 
arcj group and is interested in evaluat- 
ing^ the stimuli, ue. the test items, we 
have the problems of item fa^ysis. If 
the S^ represent tests^ then the Rjg are 
test scores, and W'C might wish to deter- 
mine the number of intellective factors 
represented in the Sfet of tests. In this 
ease we have the^ problems of factor 
analysis. As is. wefl/ knowHi matrix 
theory has been useful here in* deter- 
mining the minimum number *of factors 
needed to abcouut for a given set of 
tests. 

If tlie stimuli are toncSj or colors, of 



g=m stimulus would then be designated lights, the persons are regarded a^ con- 
Rig. Table 1 illustrates such a matrix, stituting a representative normal 'group, 



which can represent a large number of 
different sets of psychological data. For 
examples in a psychophysical experi- 
mentj liminal stimuh may be used, and 
the Rjg is a 1 or a 0 depending on 
whether the person re^onded, or did 
not respond to the stimulus*. Instead of 
being a simple yes, or no response^ Rjg 
in the matrix representing another ex- 
periment m'ight be the reaction time in 
thousandths of a second of each person 



and the scientist is interested in scaling 
the stimuli, we have the problems of 
psychophysics. In the method of Paired 
Comparisons, the Ss would be "pairs 
of stimuli," the persons would judge 
*Svhich is the larger of the two," In this 
case a matrix of l*s and 0 s is the basic 
matrix of experimental data for the 
method of paired comparisons^ and the 
law of comparative 'Judgment* In study- 
ing attitude statement or sensation for 
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■example, if one nm the method of soma^ mfUhemnto co^^^^ 



triads for determining interpoinf dis 
tances, 0ien another type of matrix rep- 
resentation as usGcl in multidimensional 
/ psyi^tophysics gives the solution for^the 
[^i^mftisionality of the sjet of objects used 
y^ jirWxB experiment. ^ • 

*^lfceufeimary M the foregoing, we may 
flaymer^re'a large number of inipor- 
Vptrnff'^sychQlogica which in- 

• ^folve^^tl^^uantirtcation of quallthtivc 
riiaterial. These prdilems in whieli ";t_ 
Jg j^ll a matter of judgmen| and juJg * 
Ments disagree" are the problems whjch 
- haA^e been classifled as psychometrics. 
or psychological measurement* Hav^ig 
indicated the nature of this 'field Jet fis 
consider the training needed for sii^ 
work, i ' ^. 

Adequate trainii>g for research in 



make it possible to obtain the necessary 
training in a shoi;ter tij^^ian the cur- 
rent arrangement does^ F^^or to such a 
reorganisation of tlie mathematics, cur- 
riculum, which may take several years 
(or even decades), it would seem that 
the student of psychological measure- 
ment should hfive the followjng courses 
now in the college mathematics pro= 
gram. , ^ ^ V . 

First yciu' — college algebra, [Uialyti- 

eal geometry, trigonometry 
Second year— differential caloulus, 

integral calculus '% _ ' 

Tliird year (one term)— differential 

equations, . % 

I'ourth year (one term)— matrix* 
^ theory ' ~ v * 

^ Probably a person^with a foundation 



psyohologieul . meiisurement can bo In mathematics sim. nr to the'ono indi=. 



grouped under five major headings 
A. Basic training in Psychology, 

This training should include work in 
experimental psychology, soeial, tlico- 
retical, compavative and ott^cr areas of 
psychology. The purpose of this train- 
ing is to give the person a good knowlv 
edge both of the eKperimental proce- 
dures used in psychology, and of the 
present s|atus of ^various psychological 
prnblems. 

Depending upon the division of 
labor among departments in a given 
university, and the interests of the stu- 
dent, such training might be obtained 
in a Department of Psyehology, a pe= 
partment of Education, a Department 
of Social Relations, or in some other de- 
partment, i 
B. Basic training in mathematics. 

The problem of precisely wliat matlie- 
mutics is most valuable cannot be an- 
swered with finality. Probably different 
people shquld study different areiis (jf 
mathematics in order to see which l)e= 
comgs more useful as various psycho- 
logical problems are explored , by psy- 
* chometric methods. It is also possible 
that a re-arrangement of the contents 



caled above would be .adequately pre- 
pared to explore various other speciab 
ized mathematical topics which seemed 
to be of special interest. It might be 
noted tliat s«ch a set of courses would 
probably iK)t ordinarily bo regarded as 
u major in mathematics. It %vould per- 
haps eoustitutc a strong undergraduate 
minor in rtiathematics, s 
.C, Basic training in modern statistics, 

fit should be recognized that the con- 
tributions o{ statistics to psychometric 
problems are essentially different: l^om 
tliose of ma til em a tics. One needs the ^ 
matliematics in order to formuhte the 
scientific laws. Statistics is concerned 
'with detennining if the data and the 
equations are in agrecmient or in dis- 
imreem^ent. Probably the student should 
have two years in statistics , including 
such topics as correlation theory, multi- 
ple and partial correlation, analysis of 
variance and co- variance, various sig- 
uifieance tests, confidoiicc ^mits and 
ustimation procedures, proba])ility 
tlieory including stochastic processes, 
and certain topics in multivariate sta- 
tistics such as discriminant analysis, and 
canonical correlation theory. These 
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techniques should be known at the op- 
erational level so that the person is 
cohipetent to apply them^ and also is 
qualified to select the technique appro- 
priate to a given problem. Insofar as 
^possible the theory on %vhich the sta- 
tistical procedures are baseds ahd by 
means of which they are developed 
would also be known ^ince part of the 
psychometricians problem would be to 
develop, or at least to initiate the de- 
velopment of new statistical techniques 
appropriate to current problems. 
D. A set of courses in procedures of 
psychologi^l measurement should be a 
central P'^^t'feA the curriculum fo psy- 
chometric trSning, There is also the 
further problehi of differentiating the 
artifacjs^ of the ^measu|ing^^ techniques 
from the scientific findings. This prob= 
lem arises in. many other areap, but 
perhaps not in the same degree as in 
psychological nieasur'ement. 

For examplei work on the physi0logy 
of the cortex has involved the use of 
opera tivej fixings sectioningj and stain= 
ing techniques. These constitute an aren 
of study *in themselves * Those hot* 
thoroughly verspd in ^uch ^hniqucs 
will frequently confusC artifacts of tlie 
technique with scientific findings. 

Correspondingly^ studies of corticul 
potentials, or brain waves, involve com- 
*plex electronic" equipment* The prob- 
lems of properly designing and hand- 
ling amplifiers, oscilloscopes and other 
equipment frequently seem= to eclipse 
the functioning of the corte^ which the 
instrHments are supposed' to record. 
However, it is essentird to know= wlicn 
one is recording cortical functioning, 
and when one is recording Various arti- 
facts produced by the eqtiipmenL ^ 
- A similar situation obtains when the 
research involves the techniques of 
psychological measurement. Almost any- 
one feels that he. can frame a poll ques- 
tion, administer it and draw conclusions; 
can construct a test of a few items, it 
devise an attitude scale ,and draw valid 



conclusions without exercising ^ other 
than a few tommon-sense precautions. 
T^re is perhaps, a great deal of ^mis- 
leading %vork in th^se areas due to the 
fact that the propcvties of the measur- 
ing instrument are misinterpreted as 
properties' of ^le phenomena sup- 
posedly being measured. 
^ During the past fifty .^years' an elabo- 
nite body of theory has been developed 
to deal adequately with psychological 
measuring in^twrnients. . Probably a 
minimum of three one-year graduate 
levef courses would be necessary ,to 
cover such material. ' 

1. A course in tlie theeiFy of tests, in- 
cluding work on reliabilityj valid- 
ity, error of iheasureiueuti adjust- 
'munt of time limits, problems of 
group lieterogeneity, of scoring, 
scaling, and weighting methods, 
procedures for improving tests by 
item analysis and item selection, 
tlie best rnetliods of eoml)ining^ 

i tests iiiio a battery, etc, , 

2. A second course would deal with 
rnetlinch of factor analysis. Matrix 
theory would be a prerequisite for 
such a course, or included in the 
ea^ly part of the course. The 
course would deal with the theory 
of factor analysis, methods for 
manipulating large matrices of 
test scores, and also with the vari- 
ous uses of factor analysis in fields 
otlier than tests. — ^Fields such as 
abnormal psychology, senso^ 
psychology. Perception, Learning, 
etc. ^Probably it would also in- 
clude some introduction to mod- 
ern higli speed ^ electronic com- 
puting proccdrues, 

3. A tliird course would deal with 
the psychological scaling proce- 
dures, including the logical foun- 
dations of such measurement, the 
experimental and logical distinc- 
tions bet%veen ordinal, extensive 

j and other scales^ the distinction 
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between fundamental, derived aiid ' nature 'of ^uch models and the func- 
tions which they serv^ in sciencef For 
example^ the deri^rttion of the orbit of a- 
planet, .or th^ trajectory of a*projeetile 
from Newton's laws of' motion might 
constitute appropriate infroductory ma- 
terial for such a coiuse. One might also 
utilize some illustrative derivations and 
theor^nis^ from field theories in elec- 
tricity and magnetismj and possibly 
some mathematical models in the* field 
of biology fVom mathenmtical biophys- 
ics. Such an introduction from mathe-^ 
rt^tical models in more, est^bhshed 
fields would serve to illustote various 
important general points^ such as that 
the basic postulates may be unverifiable 
/ directly AsometiTpes arc juerely defini- 



defined magnitures, as well as 
various giodern developments in - 
psyol^logiciil scaling. These latter 
'Would include , for example^ the 
method' of paired comparisons and 
law of comparative jud^ent, the 
» -method ' ^f successive intervals 
and the law of categbHcal judg- 
ment,K.the method of similar re-_ 
actions^ scale analysis^ latent 
* structure analysis, and the' multi- 
* dimenjional procedures in psy- 
chophysics. ^ 

Such a set of three one-year coiifses, 
if compactly and carefully drganized, 
and if the students enter them with a 

good mathematical preparation ^uld ■ . > . v , ^ ii u 

- i V . i ' 1 '1 ui tions or tgrms, and, may occasionally be 

cover the basic material now available r i , i- * 

, , . , 1 'contradieted ' by ordinary experience* 

in psychological measurement and ■ * 

could give students a fotindation so 



tliat by means of independent work he 
could move ahead/ and keep up with 
new developments ^n the field, a^ well 
aS make cQntributions of his own to the 
area. 

E, The Development and the Function 
of Mathematical Models, 

Since in the present state of psy- 
chology the development of m_athemati= 
cal models for various areas is a some- 
what new and undeveloped ideaj it is 
desirable at present to have a course 
dealing explicitly with this area* Even- 
tually a mathematical freatmcn^ will 
probably be a routine part of most psy- 
chology courses, however for the next 
five to' 20 years, such material will 
probably Im covered in it scparulo 
course^ if ^fcred at all. 

This course on mathematical models 
should include'^ a survey of models al- 
ready, developed in various psychologi- 
cal areaSj and practice in the develop- 
ment of new models. It sliould begin 
with some consideration of wclhestab= 
lished mathematical models in other 
areas in order that the student could 
become acquainted in detail with the 



It might also be desirable to include 
' some attempts to verify certain theo- 
rems experimentally in the laboratory^ 
in order to demonstrate how easy it is 
to set up a poorly designed, or a poorly 
controlled experiment and thus "dis- 
prove" some of the basic theorems of 
physics* Doubtless many elementary 
physics students have "demonstrated" 
that falling bodies do not obey the law 
of gravitation, for example, 

The fact that experimental condi- 
tions must agree approximately with 
the assumptions of the mathematical 
model is another important considera- 
tion frequently overlooked in criticisms 
of present attempts at mathematical 
models in psychology and in social sci- 
ences. 

The introductory material might also 
include a brief review of differentiation 
and integration, non-linear curve fitting, 
Pearson's method of False Position, and 
graphic methods such as -rectification 
and translation of a master curve. 

The body of the course, however, 
would deal with various mathematical 
inoduls within the field of pfSychology, 
.^^lese might^include illustrations from 
the psychology of learning, as de- 
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veloped T)y Thurstone, Hull, Bssh, 
Mostelier; E^tes and Burke. Others 
would be from tiie^eld of social^ be-- 
havior as • illustra^d by some m the ^ 
work of Nicliolas RasKevskj^^ J, 
Stewart, and G. K. Zipf . A treahnenr of 
epidemic theory as developed ^ by 
Lowell Reed and otliers might also be 
interesting 

The minimum necessaiy work in this 
area could ^probably be covered by" a 
one=year course provided there were 
tvv'O prerequisites for this course, . 

1. Differential equation's 

2. Sonqie kno%vledge. of experimental 
' psychology, such' as" a course in 

the psycho lofy of learning. 

During the latter half or third of tUis 
course, students ^ should be giyen an 
opportunity to develop new models ^ 
which might be tested experimentally. 
F. Miscellaneous S pedal Courses, 

It might be that one would want to 
consider different special courses which 
would be particularly useful fur stu- 
dei^ with special types of interest in 
Psychological Measurement. For exam= 
pie, time series, atito-correlation, or 
spectral analysis might be topics whieli 
one would wish to have available and 
to encourage some students to take, but 
which would not be particularly sirlt- 
able to requirAfrom all students in psy- 
chological mealuren^nt. 
With such traiiWng ill: 

A, Basic psyehology, 

B, Mathematics 

C, Statistics^ 

D, Psychologieal Meusuroment 
Theory, 

E, The Development of Matheuiali- 
cal Models, and 

F, Other Special Courses, 

it seems to me one would have students 
who would understand current Psycho- 
logical problems, be equipped to utilize 
measurement techniques in various 
areas where such methods were rele= 
vant. 
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All of tliis nTimrial could not be cqy= 
ered as a three=yeaft program of gradti- 
ate triUiiiug, However, if ftie studonts 
began ^uch work as uii iindergraduatc, 
this prograilr allows' a great deal of 
'freedom for other eleetives and even - 
for; another major iiitercst. In terms of 
one-year courses at =either griiQuate or 
undergraduate level the progWim may 
be summarized as follows:' \ 

\)iu>year course 
ur equivalent 

Mathematics \ = .s 3 . 
Statistics 2 

Psychology (suy) .\ G 

'Ps^cbological 

Me^isurement = . 3 

Mathematical Nhxicis ... 1 
Special = 1 ' 

Total iuj^rogram 16 

Assiuning four years of undergradu- 
ate and three yours of graduate work, 
(the la^t of which is spent on a theses) 
gives Ax- years ot ntudy. Assuming a 
load of iour courses which is usually 
not conHidcrt?d heavy gives a total of 
twenty four nnc-ycar courses in thte sLx 
years, This program specifies two-thirds, 
leaving one=third of the students time 
free for otliur sorts of^ w^oilc. There 
would thus be considerable time for 
Htudy in rclatc*d fields of special inter- 
cHb to each studeut. ^ . 

Another way of looking at the total 
load is to say that a student taking a 
PhdX in Psychology at Princeton and 
spcciali/.ing in P.sychoniutrics( provided 
he had a strong undergraduate mjnor 
in mathematics and the equivalent of 
an elective coiirHC or two in Psychology 
and Statistics) would have essentially 
the program outlined here. 

In order to obtain wolhtrained per- 
sons in Psychological Measurement, it 
is necessary tliat superior high^school 
students and college freshmen be in- 
formed regarding the nature of the field 
and the opportunities it offers so that 
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they may direct th^ under^aduate 
work in diia direction if fidid is of 
poiiible interest to them. . 

It is taportant to "erriphaiize that 
such undej^aduata fraining in mathe- 
matics and statistics do^ not constitute 
spe,0alimtio^ but ^tead constitutas 



^ good foundMon training fdt enhance 
into a number of differeht fields, in- 
cluding botii physical sciences and so- 
cial sci^GCs. It is highly desirable tliat 
this fact be brought to the attention of 
superior high-school students and coU 
lege fre^hme|ii 
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llie Teaching of Educational Measurement 
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SUMMARY OF DISCUSSION 



.Dr. Curttotip Chataian of dis- 
eu^ion, evened wltfi^two commentsi 
one QOncemed tiie apparant limitad 
population mobility of people trained 

i in educational measurement— people 
trained m the midwest, for example, 
app^ently do not mi^ate to New Eng- 
land. The other comment concerned a 

-rAther si^ifleant bccu^bnee at his in- 
stitution: A ooUege freshman who was 
doiftg'^ poorly in two of his oourses was 
asked by the counselor about his dif- 
fleulty* It appeared that he was both- 
ered only by those c^sel in whichr 
essay exams were ^ven* He had never 
had that kind of exam before reaching 
college! 

Dr. Davison emphasized die value of 
first ^alyzing the methods of teaehing 
a subject befofe trying to test it; this 
is a fruitful approach to, the problem 
of relating t^t content to future objec- 
tive* Dr, Noll agreed that the gap 
between the fprmulation of objectives 
and the test item itself is a major 
problem. 

Major Carlson, . commenting on Dr. 
Kvaraceus" presentation, noted /that 
people conducting'^ courses are often 
av^se to measurement, and suggested 
that futiire instructors shoiJd be taught' 
what they can get from measurementi 
He explained a te^nique used at the 



Air Uriiversityi after students have 
taken a tests they are sometime pla^d 
in the role of insbuctor and -'asked 
how they could improve the test, or 
what use, as an insfructor, tiiey could 
now make of it. Dr. Kvaraceus pointed 
out that a femial coittse in measure^ 
ment might not answer the needi he 
would prefer a "functionar analysis of 
failures" and continupus working out 
of appropriate #^aluation devices with 
teachers, rattier than relying on short- 
term consultants on evaluation meth- 
ods after a curriculum is designed. In 
response to the qu^tion whether it is 
desirable to have students consfruet 
their own examinations, Dr, Noll felt 
that in general students would not 
make up good enough t^ts for evalua- 
tive instruments but tiiat the experi- 
ence of making up^^ests could serve as 
an instructional aid. 

Dr. Schweiker commented on the 
problem of conflicting attitudes and 
experiences among faculty on toting, 
which seriously hamper efforts to ta- 
troduca better evaluation. He cited 
the principle stated by a humanities 
faculty ^oup Aat ''anything that oan 
be measured is not wprUi measuring/' 
Dr. Schweiker suggested diat an analy- 
sis of the examinations of such a group 
would be most enlightening. 
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Tdhe Interview as an Evaluation Technique 



1. Lb WELL KELLY 



AN EVALUATION OF THE INTERVIEW 
' AS A SELECTIVE TECHNIQUE 



I. iNraoDucnoN 

In rm broam§t sense, an miemew is 
no^ng mof a than a wnvttsation be= 
twem twO'tadi^dduabs directad by one 
of Am, toward a spedfle end or pirr- 
pOsa/The particular purpose served by 

of th© situation in wUdi it is being used 
and tiia inter\^ew^'s dgncaption of the 
possibilities of Ae mtiview as an ap- 
propriate procedure in tiie iituation* 
For ^cwripWi an ittt^ew may be di- 
rected primarily at diciting information 
- from intOTdewee* The information 
may be d^ired as a basis for making a 
decision regarding 4e intarviewee or it 
may be coUwted and pooled with par- 
^ ailel ittformation elicited from other in- 
t^viewe^ to provide tables of norma- 
tive data for a large ^oixp. In otii^ 
interviews, the purpOie may he pri^ 
m^y Aerapeutio or educational^ i.e., 
^rri^ out for tiie purpose of making 
dianges iri ttie inter^dewee, 

Regardless of the pmpose of the in- 
terview, it may also vary on another 
importaiit dimension, die degree to 
which it is stoictured, At tiie one ex- 
" treme, it may be almost completely 
staiicturedi in which case the inter- 
view&*B task is ^sentially that of orally 
administering a qu^tionnan^e and re- 
cording *fee r^pons^ given. At the 
otiier end of tiie continuum, tiie inter- 
view-may ba atoost wittiout structure — ; 
Ae tatervdewer^s task is simply Uiat of 
encwwa^g Ae interviewjie to talk* 



ThmB is no doubt but ^at the int^- 
view may be used effectively to ^dt 
attain Idnds of information from hu- 
man subjects. 'Riere is also considerable 
evidence to suggest tiiat the int^ew 
may be used as a basis for rwonably 
valid appraisals of certain personality^ 
variable. As a psyehoA^apeutio tedi-- - 
niques %e intetview has no rivd, al- 
ttiou^ parratiietically it shotdd be 
noted that evidence for its validity in 
tiiis domain is Kcto^ely scant . 

II, The Selection Iktebv^w 

I have been asked to disciiss the in- 
terview as a selective technique* Even 
wiUi this limitation, howeve-, tiii topic 
is a ^road one and also complex. 

Without doubt, the int^iew is the 
oldest and Ae most widely iised of all 
selection techniqQ^, Crlssy in a re- 
cent paper (2) notes tiiat it continues 
to be the most popular personnel se- 
lection method in private uidustry: 
Swenson and Lindpen (11), in a sur- 
vey of Minnesota industiies, found ifeto 
rarJc first among personnel selection 
procedures. And as a reminder ^at the 
use of the selection interview is not 
limited to indusbial settings, Sfalnak^ 
and Eindhoven (10), in a survey of 
medical sdiool requirements, report 
that applicant are requ^ed to report 
for a selection interview in 53 of die 
80 medical sdiools. The 27 remaining 
ones only urge die applicant to appear 
for a selection^interviewl 
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pmm^ r^^nsiU^ {ot ^@ id^ctira in- 

pmonality md ta typ^ and fui^unti of 
tirais^g for ud ©^crien^ M tatm- 
viawtog. Th^, diD peatly ^th 
r^piM to partiaulu ^@ ^ Mte- 
vilhv It^^a^um ' mplbyMr Whtr ^ 
may wb sbA to flnd ^ ocmmi^dity 
among^Varioui mm and^ propOQants 
of sd^tivo int^^w? I mggmt 
two areai of coznrounali^: (a) acom- 
moEi^ function or da^tion of the tesk 
andilb) a shar^ (eonfldence or belief 
ia Ao ^^dity of tiie tfdmique as uied 
in looal situation, ' >- 
Common FunMow In d^oibing 
what se^fns to bo tiie ^mmon function 
of tho ielectfon int^view wo ihaU ako 
d^Bn©^ it for tibe p of tibii, disoiw- 
iioni Reg^cUeis of tiie iattingp the 
cAaract@r^ti^ of Ao intwview or of 
the tecluiiquas used, die interview ai a 
id^tion technique involve: 

1. A situation in which a limited 
numbcf of persons horn a larger 
number of candidates are to be 
selected for available appoint- 
menb^(]ob» position^ scholarship, 
or ^oup membmhip). 

2. An assumption Aat individual dif- 
ferences among the candidate are 

^ correlated succ^sful per- 

f onnance ^ the role to %hich Ae 
candidates aspire. 

3. A ^nversation between an intor- 
viewer and ^ candidate for the ap- 
pointaient. 

4. An asi^sment of each candidate 
by an interviewer leading to a 
prediction (ei^licit or implicit) 
regarding ^@ probable ' relative 
success of ^e ciuididate in the 
^p^onnance situation. Wus pr^ 
dictibn may be implicit and esr 
pr^s^ in categorical form, e.g., 

. "aec^t" or "reject" or it may i^- 
_ : yolvs . a more ^licit . , rating . of 
probable suc^^ or failure eith^ 
in adjectivd terms or on a rating 



scale. It if not me^tial here to 
diWn^uA between Aose in- 
^ s^iM ha whidi the prediction 
(implicit or-@^licit} is made by 
, ^e same p^son who doii ^e ki- 
' tervim^g or by a s^ond peat^on 
who bas^ his prediction on In^ 
ioimation or a pmonaUty ap^ 
p^isal Rowing out of die int^- 
view. In either instance, die use 
of the inttt^ew as a selection 
technique involves die prediction 
of future behavior of Ae candi- 
dates widi r^pect to a sd^ted 
mt^on* 

Confidence in €ie Validity of the Se- 
lection Interview: 

"Hie most elbquent evidence of thtf 
widdy ihmd belief in, the validity of 
the interview as a selection technique 
is its continued widespread use; in most 
situationSj ths confidence in the tech^ 
nique is so hi^ diat neitiitt* die inter^ 
viewing stoff nor administfators of die 
organization even consider die desira- 
bility of dfitermining the actual validity 
of the technique in die local sitinition. 
Furtliermore lack of evidence to sup^ / 
port such eonfldence in the validity of 
die selection interview do^ not keep 
intelligent people from believing in or 
even testif^feg to such beliefs, llius wie 
find the authors of ^sB&smmt of Men 

(7) making die bold stetement diat 
"the interview is probably tte b^t ^d 
only indispensable method of ass^s- 
mfint." In a sunilar veinj Alec Rodgar 

(8) in a recent article entided ^TThe 
Wordiwhilen^s of die Interview" states 
"the interview is the itandard means 
whereby p^ple are judged for m^y 
pu^oseSj^ and is likely to remain lOj 
maybe till the end of time." 

Further evidence of die hi^ Meem 
in which the selection int^view is held 
by psychologically frained pe^ons d©^ 
rives fr^a di^^ Michigan A^s^sment 
J^oject (B) on the sd^tion of cUni^ 
psychologists. In diis proj^t, w© used 
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two intarvlaws as parte of a wedcJong 
sBTim of ass^sment procedures which 
'Included also an extensive battery of 
objective t^tSj a battery of projective 
testip several situatign t^tSj etc. Near 
ft© end of our major assessment pro- 
^am^ each staff m^ber .was asked to 
raiJc ^der tfie proeeflures used in tenns 
of fteir usefulness to him ifl arriving 
at judgments concerning the future 
perfonnance of the candidates. Practi- 
cally all of the staff of 25 persons 
raiAed the interview either in first or 
second place. 

-^111. EVIDENGE ReGABDING THE VALID- 
ITY OF THE Selection Interview 

The selection interview may be a 
valid technique of personnel selection 
in some situations. However, as I think 
most members of diis audience know, 
the accumulated evidence is such as to 
throw tiie burden of proof on the pro- 
; ponents of the methodrNearly 35 years 
ago Scott (9) and HoUingworth (3) 
in independent pioneer investigations 
reported surprisingly low interjudge 
reliabflities ind validities of interview 
judgments regarding sales a'bihty of 
prospective salesmen. Since Uiat times 
k many comparable studies have been 
conducted and most of the research 
findings poinf to similarly low relia- 
bility and validity of interview judg- 
, ments. Occasionally, if one searches 
long and hard, he can find a study with 
somewhat more promising results. One 
such is Vernon and Parry's (13) report 
of a war- time research on the selection 
of jtrainees for commissioned rank in the 
Royal Navy. Pr^ictions were based 
first on a series of paper and pencil 
^ cognitive tests; second,^ an interview 
carried oiit by a conventional officer 
selection board which had access to the 
test results and also to reports by com- 
mandihg ofiicers on the candidates; and 
third, an interview by one of three psy- 
chologists who also had access to the 



test results but not to the reportSp In 
this study it was found that the ^gni= , 
tive tests alone wece better predictors 
of the criterion than the judgments ol 
the, officer selection board, but the psy^ 
choibgfats did somewhat better than 
either. Similarly Bobbitt and Newmann 
(1) report promising validities by in- 
terviewers in the predicjEion of success 
of candidates in the officer ttaining 
program of die U. « S. Coast Guard 
Academy, In this study the combined 
judgment of two^ interviewers co^e- 
lated with the pass/fail criterion .49.. 
This is most entouraging until one 
^otes tiiat the test scores alonCs which 
w^e known to the interviewerSj cor- 
related .47 with die same criterion. In 
this particular^ study a statistical com- 
bination of interview Judgments and 
test scores yielded a validity of .56/but 
whether these findings could be repli- 
cated with other interviewers is not 
known. Likewise, Hunf, Wittson and 
Hunt (5) report low usefur validities 
Uor even a brief psychiafric screening 
interview when Judged against criteria 
of later adjustment in Navy life. 

Such results, however, are except 
tional. Even such an ardent proponent 
of the selection interview as Rodger 
(8) admits the "general tendency of 
experimental flndings has been to show 
that even where the interviewing has 
been done by psychologists^ interview 
Judgftients can be very ^variabla and 
very, wide of the mark." Furthermore^ 
individual differences among Inter- 
viewers with similar fraining appear to 
be sufficiently great to suggest the need 
for some procedure of selecting inter- 
viewers to conduct selection interviewsl 
Let us look briefly at some of the 
ihore recent findings concerning the 
validity of the interview as a selection 
technique. In our own work on the 
selection of clinical psychologists (6), 
we. uped two interviews. The first was 
one hour longi conducted by a staff 
membfr who had previously made 
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fadgcam^ of candidate on basis 
of hia ^edtnti^ fite only« lie seoond 
hkt^viBW y/Q$ two hourg long con- 
'dupted ty a dUff^ont 'itaff mOTbw and 
^nried out*orfy afta the totarviewer 
had pr^ou^ made an inteniive itudy 
of j^L^^te's grid^tid flle^ bis 
scori#on an ^ttrtensive battery of pblee^ 
tiv© and proje^ve a biopaphical 
informatidn invantory and a long auto- 
bibpaphy. Thaie interviews were car- 
ried out by trained prof eisional persons 
who were pmriitted to structure the in- 
teiid^vi in the mannar w^idi tihey be- 
lieved most usaftil for the task at ha^d. 
The vahdity of judpnents made before 
«nd rfter each type of iritervie% was 
^dmatad agaihst a doien diff#ent 
caitmia obtainad" foitf yesfs later. The 
r^ul^ were such as to force us to coii- 
clude diat neidier of these interviews 
made an essential cx>ntribution to our 
ass^sment propam. Actuallyj tiie me- 
dian validity of tiie judgrnents made 
after each type of interview was only 
.01 greater than Ae medmn validity of 
jud^nents made before the interview. 

At least tiif validity of diese inter- 
view Judpnents was not negativej as 
^was true for those reported in a recent 
study by TTiayer Thayfr, in a^ 

" doctoral dissertation at tiie University 
of Pittfbur^i attempt^ to predict the 
subsequent field success of missionaries 
to whom a battery of psychological 
tests *^ad been administered some £0 
years earliar. In tiiis studyj an ad- 
mittedly fallible criterion was predict- 
able with a ' cowelation of *53 with a 
battery of three psychological tests but 
Qie ratinp made by thf ScCTetary of 
fte Missiona^ Selection Board, pre- 
sumably based on interviewSp refer- 
ences md otfier papers* showed a nega- 
tive coffelation wiA the criterion 
measure. 

Perhaps tfie most cogent evidence for 
- doubting-" &e validity of the <^nven- 
tional sdection toterview appears in a 
recfat article by Hdt and Luborsky 



(4), whicK$reports on the Menninger 
Foundation research project on tfie se- 
lection of ^psychiattfets* In conteast to 
die Michigan ass^sment propann, the 
Menning^f project relied most heavily 
on judgments based on selection mter- 
views and a batteiy of individually ad- 
ministered psychological tests. In this 
projectj each applicant was independ- 
ently mterviewid by tiiree psychiafristSi 
each of whom made a global prediction 
regarding the candidate's probable suc- 
cess in psychiatric training at die Men- 
ninger School of Psychiatry. In many ' 
ways, Uiis study would s^m to have 
been almost ideally designed to yield a 
maximum estimate pi the validity of 
selection interview^ Firsti it should be 
noted that the i^tarviews were con- 
= ducted by staffi/psyehiabristes presum- 
ably expert iiythe art of interviawingi 
secondly, thf predictions wera made 
with T^^^^o criterion performinte in 
die local situation about which the ih- 
terviewers were ma^cimally infonned; 
finally, although Holt and Luborsky do 
not provide evidence on this point, 
many of the interview's, in their role 
as staff ^members of the * Menninger 
School of Psychiatry, must have con- 
tributed to the later criterion evalua- 
tions of the candidates. In spite of these 
seemingly optimal conditionSj the valid- 
ities of these interviewer Judgments 
were shockingly low^ the median being 
.06 for thej 14 interviewers. Further- 
more, only one of these 14 presumably 
expert interviewers had a validity large 
enough to achieve statistical signifi- 
canee* 

Such evidence does not prove that 
^ the interview is valueless as a technique 
of personnel selection. There may be 
some situations in which some inter- 
viewers are able to use the t^hnique 
and arrive at Judgments widi hi^ pre- 
dictive validity. If so, it is most un- 
fortunate Aat they have not^been. re- 
ported in tiie literature. On die con- 
trary, all evidence available suggests 
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dirt til© ta^oiquq ^apt tp havf iuffl- 
ci^Uy low vw^ddfy even und^ optimal 

jbonditioni to mske doubtfu] iu ganaral 

futility as a ieltotioa devloe. 



Wa tttf ftius fot^ to j^nebde that 
A© iBS&it widely And confident used 
t^hnique'^of p^onneLialaction is one 
for whiah tiiif # ^ iurpristogly little evi- 
d^ee of pwMMvB validity* TWi em- 
ous' situation appesu^ tb have its paral- 
lals in cUni^ field in tibe Gun^t 
popularity of projeative teduiiqu^ 
largely unvalifiat^ for predictive pur- 
pose and in the field of education. widi 
the continue wid^^^d tif e of essay 
axaminatious. In aU tivee instance, the 
choice of the te^nique is obviously 
baied 6d f actors otii^ tiian-evidenca of 
predictiva vahdityi I do not pretend to 
Imow" what all of th^e oAer factors 
are, but I have a hundi as to what is 
goin^ on* Note ttat iri each pf the tiiree 
cas^, the tedmique is chosen and used 
by professional pWion| confronted with 
the ii^Dessity of making decisions about 
people—decisions which are si^ifleant 
to &e organization ofvwhich the pro- 
f^sional person is a part ^nd/or to the 
prions about whom tiie decisions are 
made. One ^nnot take lightly such re-^ 
iponiibiliti^ as deciding wheth^ it is 
A or B that is hured for a particular job^ 
C or D is sent to a mental hospital^ or 
E or F that gets into medipal schooL 
Id^lly^ such decisions should be made 
on the basis of t^ted techniques with 
high prefflctive validity. Unfortunately, 
* as we blU knowj such techniques simply 
do not existi The b^t of our tools lead 
but fallible predictions of later cri- 
terion behaviori they enable us to guess, 
ri^t mucA niore often than not but 
i^me are still many errors of predictipn. 
Furtiimiiore, in Ae domain of psycho- 
metric testSj we have r^souably precise 
estimates of the accuracy of bur pre- 
dictioh and of the magnitude of our 
errors. And in tiie imper^nal situation 



of a selective program bas^ on piydio^ 
meWc procedures, we can tolerate the 
truth of our fallibility— knowing diat 
our errors havf at least been reduced* 
Now inany persons do not appw' to 
be able to accept tiie inevitability of 
errors of decision and prediction inher^ 
ent in even tiie b^t of om tools for 
evaluating and predicting himan 
havfor. Such persons idedisti^By 
search for an instrument ^th ^more 
sensitivity to the lubtle ndknces of hu- 
man beha^aor and personality, and 
ndting eviderices of such sensitivity in 
tiie writings of poets, drainatiste, and 
philosophers, conclude that tiie b^t 
insfrument for ^e task is anoth^ hu- 
man beihg, perhaps tfiemselv^I ^Riey 
next proeeed to try out this newly dis= 
fcovered insbunae^t* by inter^dewing a. 
few people, by inte^reHng their hand- 
writing or reading their essays, Lo and 
behold, the insfrument appears to jyork 
and work welL Smcfe die insfrument," 
i.e., die human being and die technique 
works so weU, it should^ obviously ba 
used for iome practical pilose* L6t"S 
try it out in some practical selection 
situation. And so Ae interviewer, die 
rprojecttvist or die proponent of die 
essay test go^ to work* In each c^e, 
the technique is employed by a person 
who is already pretty weU convinced of 
the validity of his tool and dius rea- 
sonably confident of ttie coirecta^s of 
his d^ision regarding individual cas^. 
Under the circumstances, it is not sinr- 
prising that the user of tibe techniqua 
rarely finds occasion to submit himself 
and the technique to a frue validity 
check. Instead, as die result of ^ch de- 
cision made (a dectoion which Just 
dares not be wrong! ) he becomes even 
more convinced of the validity of his 
technique and himself* And if someone -. 
else insists on inv^tigating the validity 
of the technique, he finds many good 
reasons why die r^ulU of ^e study are 
not to be taken seriously; for example, 
the criteria used were not appropriate 
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ior tito tisai? of te^^^ue w&e not 

' IM^i committed on^ilf to ih@ 

pbdtioQ titot tfaa humazi b^g ^ 
^ most ^^tisl part of iss^mant 
proofs It fdllom nahu-^y the 

T f Mdicri^ ttf ^^0:«i^^aii to lie imd 
wffl li€ ,dn@ wUA mihi£c^ role of ^ 
^0 human b^g who mm iU it ihould 
one whiA provM^ for WyfyiVrp? 
^obQityf om rtqufras ,tx- 

teniiva usii of good fu^pa^t/ Further- 
more, it Bh6\jld h^ Qn& whic^ prb\idas 
a TE^stavm ^^ ^iSo^W'^on to ba iate^' 
pmt@d by tfie human mted iisoo from a 
oonraipn smi^ po^t of ^tw it would 
seem ttiat tiio hmt dediioni aro ^osa 
made on basis of &e most infomfa'' 
ton* The intendawp ^^li^y the un- 

^"^sMeture3~inf<&r^^^^ in aadi 

of ihesQ raipec^ and hancep I iugg^t, 
serv^ adzn^ably to ra4uca the tiveat 
of amdety whi^ \i^uld oflierwisa be 
pi'^ent m parsons who a^ept the r^ 
^)OnaibiUty for making important de- 
dsio^ ragarding the livas of others. If 
it serves diis funation as waU as I have 
mi^^t^y it is hardly sm^rjiing ^at 
tihe intarview Is so wnfidenUy baUaved 
In by ib propontot^--e^^danQe or no 
evidenee* 

Vt Anotsr Look at toe Moslem 

The more^I think about the selection 
int^TiaWp the surar I become that it is 
a tadc which no human being qbix be 
^ eoEpectrf to wiy out* IM us 1^ at tfie 
task or ralh^ die of tasl^ in- 

volydd in the maldng of valid predic- 
tioni of crit^on p^ormanca, Wiat 
^ WPyld .tib© mt^riew^ n^^ to Imow 
and to do? 

1. He would na^ a Aorou^ knowlr' \ 
edge of ^e ptftomancas d^ ; 
mwd^ in ttia oltMon lituation. 
If a previous job analysii does not ^ 
provide iu^ ^ormatioDp ha : 
iffioffld'^ reaUy^Uy ohei And let " 
us not forget that, although die 
oitarion may ba an overall or 



global pnOp a more ^tful analysii 
is Ucely to show it to b^ a 
weight^ ddmbinatioh of uncor-^ 
related dim^isions, 

2. He would nmd to Imow ^a rele- 
vant variabl^p i,a,p what abiliti^i 
eh^aat^stif^ and traits of p^ 
pie are related to p^ormanoe on 
tfie critMon, "ni^ would include 
a Iqiowledge of whi^ variable 

; have ino conr^tlon \dth the ori- 
terion\ 

3* He ';^OiJd need to know how to 
^ sde^ind to elicit beha^or in tiie 
interview whi^ are valid todi- 
oators of the relevant pradictor 
variable, " 

4, He would need' to know how to 
evaluate or weight each of the be= 
haviors elicited ia :^e intarview 
in order to arrive 4t tha most valid 
score or rating or the relevant 
variably. 

5, After havirig arrived at Ais points 
die interviewer would neart have 
to weight or combine each of fte 
variables avaluatad during Am 
course of the interview or in ord^ 
.to amve at an overall predicdve 
"icora'' for tfia candidate/ Inci- 
dentally, it should be noted ^t 
the preparadon of this implicit 
ra^esiion equation would nacei^ 
sitate A knowledge of the Int^- 
correlation of all predictor 
variables evaluated as will as the 
correlation of ^eh with the cri- 
terion. 

Lpoked at in this way, it will be seen 
tfiat tfie selection interviewer is 
pected to combine in one pmmn the 
role of the job analyst, Ae t^t con^ 
sbuctor, the t^t admimsfratorj the t^ 
scorer, and &a statistician* Fur&^- 
morej suce^sful perfomartce would 
damand tfiat ha be at least minunaUy 
coinpetant in playing ^ch of th(^a 
several rol^* tta end product^ Ss 
judgment rag^tog the probable per- 
ionnance of a ^nqlidati^ the criterion 
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situation^ would be ^dtiated as the re- 
" suit of poor parformance in any of tiie 
saverftl roles* For matancei even though 
a pmon ihould be Hi^y competent 
at appraismg personalis from the in- 
terview, he might fail miierably in an 
rfort to predict sueceiS in die critmon 
situation by wtue of & lack of knowl- 
. e4ge eoncmiing tiie relevance of each 
of* the personality variable to the cri- 
terion perforimMice* < 

It seems w^ely that selection inter- 
vie>vefs ever think of tiieu task in this 
way* In fact, were tiiey to do so, like 
the thousand-legged worm, they would 
probably be so braumatized by the 
compl^dty <©f their task as to be unable 
to functidh at all* Furtfierinore, it must 
be remembered that most selaction in- 
"^tarviewing is carried out by pedjple 
relatively imsophisticated with respect 
to die psychometric stapi enumerated 
above. Even dioie psyehologists most 
given to, using Ae selection^ interview 
are likely to bi those less sophisticated 
\n psychomefric procedureSp And if 
confronted with the frue oDmplexity of 
tl^task, such persons are likely to t^a 
refuge in die belief that the Job , is 
really not as we have analyzed it above 
but rather one requiring 'the artistic or 
t intuitive appraisal of the total individ- 
ual by processes not completely com- 
municable and certainly not statistically 
manipulable** Furthermore, such ^per- 
' sons are likely to argue that the result- 
ing global judgment has more validity 
than can possibly be arrived at by the 
use of objective psychometric devices 
and statistical equations, Certainly this 
is a tenable hypothasis and a testable 
one* Unfortunately, as of the present 
date, 1 know of no evidence to support 

it , 

Since die task of the selegtion mter- 
Viewer, when conceptuahzed in psycho- 
mefric foOTp is clearly an impossible 
one, it is only natural for the inter- 
views to proceed on what seems to 
him a most common sense basis, namely 



ieoure as much formation concerning 
the candidate as possible, and then 
weight these items of information in- 
tQitively oft die basis of whatever 
theory or biases 'he has picked up re- . 
garding the functioning of human be- 
togi in the situation for rwhidi he is 
making die selection. The absence of 
tested knowledge concerning the actual 
relevance of bits pf information or spe- 
ciflc variables to performance in die 
criterion situation does not bother him 
too much, He mterely substitutes his 
subjective impressiws of the appropri- 
ate weights. Furtheimdre, thb intuitive 
weighting of evidence goes &h without 
reference to the interrelationships 
among predictor variables. Each item 
of information's weighted as it is 
elicited with the result that even rele- 
vant variables may be overemphasized* 
In general, I fear diat the whole sihia- 
tion is one which encourages the inter- 
viewer to use each bit of additional in- 
formation in a manner which introduces 
as much error varianc^ns true variance 
in his judgments. . 
VI, The Future of tme Selection 
Interview 

In closing, lam going to "stick m 
neck out" and make a number of pre- 
dictions^ 

First, I predict, with a very high 
level of confidence, diat the selection 
interview will continue to be a widely 
. used and highly respected technique. 
No; amount of negative evidence re-* 
garding its validity seems hkely to 
change the situation. Second,, I predict 
that its popularity will decrease only 
when and to the degree that more valid 
techniques opd devices are developed 
to do the pr^cticariobs of selection in 
our complex society, These must be 
done by somebody and in some way. 

Thirdly, I predict that improvement 
in the selection interview itself will 
come about only by fractionating die 
total task, i.e., dividing it into a se- 
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qu^oa tit ea^ of whi^ be 
im^mmhfy qamw out by- w intelligent 
and tirab^ human babig, H&ui, oAm 
J person mi^t be asilgpi^ tiie task^f 
Uiciting ^omatioD^ anotber ^at of 
Bssmning pm^ndity v^abte, an- 
ir^^^ ©Aif &t of- wii^tog the asiw^ 
variable ta an ^plrically detmn^ed 
manner to a^ve at the pr^qtion of a 

- oit^OD. In^ aU honmty^ I doubt Uiat 
any amount ' of training of the kind 
cunrently deiO'ibed in texts on parson^ 
nsl int^rvisM^g wOl enable a iingle 
individual to do wholf ieriai of 
^inp now ^cpectad of the selection 
dntarviewtt'. Sinca we give him an im- 

^ possible task, perhaps we can forgive 

- him fpt devdoping unjustifled eonfl- . 
denoe in tiie work which ha do^. Fur- 

^ ttiermore, ilnce if our malysii is right, 
he must develop tiie confidence in order 
to reduce his arikiety, we can hardly 
expect ^im to do the rescaTch needed 
to improve tiie tedinique* Such re- 
sear^ must be initiated and supported 
by perionj who believe Ae problem 
iufflciently important to justify eon-^ 
iiderable expenditure of res^rch effort 
in ordw to change a currently doubtful 
practice. The research itself must be 
"^dotie by persons ttained in the skills 
demanded by the complexity of the 
task but who are also capable of oper- 
ating in a practical world in which se- 
lective technique are' now evaluated 
* largely in terms of dieir face and faith 
validity. 
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INTER.PERSONAt ASPECTS OF THE INTERVIEW 
PROCEDURAL TECHNIQUES AND RESEARCH PRACTICES . 



Someone sEFQ^s an interview ks a 
oonvmatlon with a purpoie. The pti- 
ma^ pilose of tfi© taterview ujed as 
an evaluative t^^que is to enable 
one or mora intefvfewm to evaluate 
the totendtwa©— ^.g* in indiistry, with 
r^jr^ Ji£i Tfltti^i for a ^ven va- 
^mey. Now, in erd©f to bound tiie 
sc^pe of tiie MB^BStien% one or bo^ of 
two ff am^ of rrf^renee wre uiuJly fur= 
iii^ed ^ intafvii^m. 

1* A deswiption of ^e tasks 

pect^ to be perform^ by Ae in- 

tttviewee* 
. 2. A di^CTiption of .&e quditi^ or 

tarai^ prasumed to chwraeterize an 

incumbent. 
Intervievsdng aj a pmonnd asi^s- 
ment t^mique variei not only from in- 
ttt^ewer td interviewffl'jiJbut witiiin tiie 
same interview®' over a period of time 



ratiier than upon tiie d^ision of one 
person in the organizatipn* Tliii, how- 
ever, raises qu^tionii Are^ tiie ieveral 
intervieweri using a common firame of 
reference? To what extent do tfi^ 
a^ee among themselv^? How much 
ints'-niter^ewer agr^ment is r^y 
desired? It is my purpose now to dis- 
euss iome of tiie ramifications of tiieie 
problems and to outline briefly some^ 
ways of foeuislng interviewer trainkig 
to promote improvements in tiiis aspect 
of interviewing* 

' We would, I Ainkj hypoA^ize that 
tiie more ifructured Ae interview^ tibe 
more inter-interviewer i^etoent. This 
is in part due to the closer presoription 
of what may be askrf— in tiie exttraie 
eaie, the actual wording of qu^tions 
and sequeKoe of qu^tfoi^s are set In 
part it is attributable to the detailed 



If it fa to be used, then several prpb- nature of the rating stature witein 



Iots of botii inter-personal and intra- 
personal c^nsisteiic^ must be dealt 
witib. Most of tti^e, fortunately, admit 
of identifleation ^ou^ research and 
of at least partial iolution tiirou^ 
bainmg. ^\ 

In any but ie smaU^ organi^tions, 
sev^al pOTO^ share the responsibility 
for id^ting new mOTbd^i tiius raising 
tiie problem of tater-pifcsonal aons^t- 
enc^, I should say, parettAetioally, mis 
is as 'it Aould be* On biany countSi I 
tiiCac tbe sel^tion of a single applicant 
or ^ndidate ^ould be^ased, as a mat- 
ter of poUoy, upon co^ensus judpa^nt 



which tiie interviewer is asked to 
quantify his judgmints. If inte-intar- 
^ewer consistency were our only eon- 
cem the way to accomplish it would 
seem to be, in the main, by structuring 
the interview as much as possible* But 
liere" s tiie dilemma, if our lnter\^ewOT* 
quantifications of Judgment coirelate 
hi^y it cW meant (1) ^at we can 
accpmpfish about the same r^ulti using 
fewer interviewers (if multiple inter- 
viewing is being praetie^— eitiia^ 
board or wnseeutiv^ type)! (£) tiiat 
individual inter^ewers are not maWng 
unique conWbutions of judpneni If 



J 33 



our intendewers' quantific^tioi^ of judg-^ 
ments coFrelate low it &Ln memi (1) 
^at ^ay ih^e no common JBrame of 
rafrience for their jud^antii (2) Oiat, 
their individual judgmenti aj wall m 
tiieir concaniuj jud^aiento are unrali- 
^ Able; : (3) that md^^ aialdn|; an indi- 
\idual ^nttibutioit^variflabla by va- 
lidity procadiu'^ noted below. If the 
interviaw^i ftfa aaked to cita evidence 
substantiating tiieir jud^ant quantifl^ 
catians a (intent analysis of die evi- 
danea, interviewer by intarvdewerr Vdll 
shed h^t on tfie tanability' of the con- 
clusion of no QOmmonframa of raf er- 
enee for Uieir Judgments. If the content 
analysis is pursuad over a period of 
time, wa can datermine inferenBally 
how I'aliabla the judgn^ents ara. If little 
overlap of 'avidance contant Is found 
fifom interviawer to interviewer^ train- 
ing should include a discussion of what 
the j^j^crib^d b'aits mean and what 
QOmpr|s^ varying amounts of each of 
them. ; 

Tuniang back for a momant to the 
case of high inter-ihterviawar correla- 
tion I suspect} on the basis of scant re- 
se^ch data deseribad by Sternberg, 
that iiidividuai differencas do exist 
^mong any npup of interviewers on 
sha^n^s of ^iscriminability of Judg- 
ment, frait by frait, as well as ovar-alL 
It seems to me we lose the opportunity 
to make potentially valid use of this if 
we structure the interview so muGh that 
tiiesa individual conttibutions of judgf 
mant" cannot be made manifest. 

As a pragmatic solution to this di- 
lemma, I have espoused in my indus-^ 
^brial vyork jemi^sttuctured intarviewing. 
By this I mean arming each interviewer 
with as precis a a job dascription as can 
be obtained, furnishing him a carefully 
pr^ared sat of manpower specifications 
cou^ed in behavioral terms; requiring 
a quandflcation of jud^nent on pre- 
seribad fraits, th^^ too, behaviorally 
described^ as well as a citation of evi- % 
dance to substantiate the Jud^ents 



madep However^ Aa interviawer asks 
what questions ha deems desirable in 
order to aixive at his Judpnente. Con- 
tinuing research, along the lines to 

. which I have made raf^encc, I view 
■ as essantiaL 

It is outside Sir scope of my assignad 
topic to tteat intr^-interviewer consisj- , 
eney. Obviously, in tiie over-all design 

^of resaarch on reliability, provision 
would be made for exploring this aspect 
of the problem. A facet of infra-inter* 
viewer consistency comprises sporadic 
and systematic arrQrs of parceptual 
process, attitude, and Judgment, Two 
former graduate students of mine, 
Regan (7), and McCandhsh (5)s have 
concerned themselves with research in 
this area* 

Now then, if the interview is to be 
used as an evaluative technique^ what 
of its vaUdity? 

It is not my purpose to get into a 
general ^discussion of validity. I should 
rather like to point up some ideas 
which may properly be subsumed hare 
and which have a direct bearing on my 
^assigned topic of inter-personal aspecte 
of the interview. I should state at the 
outset that I "believe it is more appro- 
priate to speak of the validity of the 
individual^ interviewer s Judgments than 
it is to talk about the validity of the 
4nterview^As a case in point my at- 
tention w^ called tg a paper at the re- = 
cent PariB^eetings. Husen (4), the in- 
vestigator, presented 'data not only 
concerning the validity of individual 
interviewers' judgments but the effect 
of age, sex, trainingj etc* on such judg- 
ments, ' 

In passing, w0 . should rein|nd our- 
selve^s Jhat the unjique nature of the 
interviAv among screening and selec- 
tion teehniques raises an immediate 
issue of whetherdt should ha conducted 
independently of any khowle^ge of 
data from tests, references, application, 
etc., or whether it should be an Inte^ 
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pative st^, TWs is ft moot point in our 
prmmt sbit© pi l^p^rl^ge. However, 
when, it tm used totapatively maos^ ob- 
vious aompUeations arise in tihe deter- 
mtoatioq, of validity* Many inv^tiga- 
torSj iBduding ^taen* have shown that 
r^tively Httfe a* added to ft© predic» 
tion when interviewer Judprients are 
made intepativelyl It Ib not within the 
scope of tiili pappr to discuss Ais issue 
but it shoidd be bpme in mind In what 
foDows, 

As a &st note under validity^ it 
would mppwr to be good practice to 
apply tt^ditional correlational analysis 
to the individual interview^'s^. jtidg- 
l^ent quant^cationsj predictor var^a- 
blesj and such ifrtervieWfee perfoflmanee 
data las _ m|Q^.^ be ^stibsequenU^ ^^^^ 
avaUaUe, criterion variable, Ilie dffl- 
culty of course is that a pwticular in- 
ter\*iewer may not do enough interview- 
ing to yield a sufflqient number of 
oases for tiiis aft^ro^. Certainly it 
would be a" rare situation where si^- 
cient data of ttiis kmd were available 
for each of flie several intervleweri. A 
poof substitute for this kind of validity 
check IS to use a "position^ offered— 
position not offered" dichofomy as a 
quasi'criferion and to anilyz^ the judg- 
ment quantifleatioiii of &e, ^dividual 
int^^iewers against this nimsure. * ^ 

As a second point under validity tl> 
should like to mention the place of f aq- 
tor analysis* in reseMch on the intet- 
viw. Sternberg (8), Barnes (1), and 
more recently, Chillian (3), graduate 
students of mine, used direct factor 
analysis on Interview data. They factor 
analyzed matifiees obtained by*intercQr-, 
relatiAg ti'aits on which interviewers' 
had'ma^i' quantifications of jud^entsi 
St^dig^^if Ais type help in sheddjngv 
li^t^ lipon tiie concepts underlying the 
s^,fi£dal judgments made by the in- 
.^^i^pTB. It would be interesting to 
i' db tfi^ idn4 of shjfdy on individual -in- 
^ terviaw^V data, For example, would 
the data toin a "good picker" yield the 



same factor eonsftuct as those from a 
'*poor picker? v % 

Cash (2), another former student, 
fac^r analyzed a matrix obtained by in- 
tercorrdating applicants for^^ie^ same 
vacancy. For factoring such a patiix of * 
intw-pariopal correlationsi the mosf 
paAimorfous hypothesis, he reasoned, 
shotild be tfie existence of a single man- 
factor-type. He found tije hypodiesis 
untenable. Furdier, he found Uiree ^ 
man-factor-^p^s whi9h he then at» 
temptfed t& iaentify by means of subse- 
quently collected omth^Job perform- 
ance "data. This type of study has 
potential value in defining Ae mom 
man-power requirements for specific 
openings. Also, a similar d^lgn mi^t 
be used with on-the-job performance 
data to shed li^t on "t^es' needed-- 
to fill Job needs. • 

Where multiple interviews are ac- 
corded applicants and where many in- 
terviewers are used, it may be eidight- 
ening to do an inter-personal type 
factor analysis of interviewee^ Il^^ht 
be found tfiati>several iriftf^ew^r 
"types** existed, jjfi such were ^jie^ c^e, 
it might provi^ a basis for setting up 
combinations of interviewers to maxi- 
mize discriminability of judgments frait 

by trait. i t 

.For my third point under^ validityi I 
% iriust refer back to thgjise of content 
■ ' analysis mentioned e^lier* It seerns to 
, me/that both reliabihty and -validity are 
likely to be increased when interviewers 
have clearer concepts of the traits .or 
qualities which they are frying,^, judge. 
Content analysis seems to be a tech- 
. . nique which is useful in clarifying these 
■ concepts. What shall we content ana- 
. ?lyze then? 

' ^ 1. The evidence cited by interview- 
' f ers to support their judgments, as 
: previously mentioned. . 
' 2v. Definitions of tiie traits formu- 
' lated by interviwei^* — 
3,^ Sound-scripts of interviews^ , 
/ 4/-Quditative statements mad#ftbout 
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c^didat^, in meriioSj etc* IncP 
.dantaPUyj'^i^ a by-product of supK 
ai^y^ ' tnfereijcei : cap be in^d# 



,|3jcturc;^a|; the' candidaje. .;H^ ^buld 
>have iO^knowWge of the nature and 
e^fent . o£ individ differenbes, Ha 
jliou^ be trained in interviewing tech- 




lntmdw«*i judg^Ait ^pon o^^^ /nj^es^and speeiflcally in thpse used by 

lirc^ of sflection data^ f ,gl test^ 
tmplUi ebDeg©' reedrA ' jPor b|- ' 
stari^, tfia wfi^it eneouritered^an;. 
ifitayvlewer t^^hp cited; little .alrt 
weapt rank ^ candidate in theit 
graduating claii to iubitahttQta 
his jud^ente of Aek; QraJ .Ex- 
\ pression. < ; . , 4 

As a fourth aspect of vahdi|% 1;^ 
should like to mention 'somediing kfa* 
dred to contahtf analysis,, nan^elyr an ■. 
analysis df evideripe chaek-pfi llsis .if; 
such exist, A graduate ftudent of niinfij 
Wakad (9)* is pi^eiently mj^ng. ^a,. 
kind of an analysis of the same cases as 
were included in Chilliaus stud]i men- 
tioned earlier. It seems to me th^re aye 
two usafi^^vays of doing this. 



% BjAAalyzihg tiie : evidence :itatns 
! as you wbiild^- test 'items against 
internal and, il available, ^xtemal 
criteria', ^ ;^ : 

2. By seeing if pod pick^s" show 
different pattelrns of checked cvi- 
dance than do "poor pickters " 
Unfortunately Wakad's data are 'limited 
to ( 1 ) above and are at this ptage in- 
complete, it ^ 

Now in order J> to mave reliable ^and 
valid interviewinji idonCj whaf should 
the interviewer bidw? Along what lines 
should' ha be ttWifdP certaihly 
should have an ilitiWate knowledge o{ 
the job to , be donb and die man-ppwer^ 
fieritioned earlier. Iii J 
*^bW informed on the^ 
acture, management 
pr|ieticeSj' and the 
so 0ome knowU 
whom the ir|- 
ive to adjust' if 



the dompany. This shoi^ldp of course, 
-inchide acquainting th^ intpr^dewer, 
\ with form^ used and the proc^uml 

manuar if spch exatii 
■ - What 4re some feasible toining 
VmethodsP Booki, pamphlets, and re- 
vl^rints bn individual d&arences, screen- 
ift^and selection tools and procedures^ 
; on intarviewin^techniques should 
^l^e available. These are more hkely to 
^ be uSed if a reading reference guide is 
pifepqred, poi^^g up these materials, 
Lect^e4isci^^s by specialists as 
welt as experienced operating pe^onnel 
= coriiprjse a -useful sipurce ^ inftenatiohi 
; Bola-playihg jesjsions !niay be con- 
ducted^ dujing* which e^eh participant 
takps bdth fiifi role of the interviewer 
. and: that of • iritajyiewSe. "Do's"^ and 
*'dop*t^s" are demfiristfited dramatically 
'by /thi| means.^ Aitual eases from the 
persdrinel files can ' be used refrospec- 
tively to high-hght-areas of success or 
failure in property assessing candidates. 
Also, in tfiis jcorinection I have found 
play-backs Of 30undscripfs particularly 
\helpful. A§ mentioned b|foi;e, oral or 
written "*?eed-back" of research flndings 
augmented by conference metliod dis- 
cussion is . another toining aid that has 
value. Finally, the more the interview- ' 
ers themselves participate in modifying 
procedures, changing formSj revising^ 
\manuals, the more they are likely .to 
learn and the mPre seriously they may 
.handle their re&pohsibility for doing the 
best Job of interviewing they know ^ow, 
Incidentally^ Husen's paper, previously 
cited, contains data 'to substantiate the 
contention that improVCTient in judg- 
ment can be accomplished 'through 




specifications as , 
additiotii ha shoul^ 
organizayonal sta 
policfS, compari5 
like. HI should m 
edge of the parsons 
tarviewee is gofi^g to J 

b© i^.selectad*^e..^^uld also be ae-,^. training. . 
^quainted with oths ph^es of the eyal-Hr s Time limitations prevent a discussibn^ 
Wtive procadiira* ^p&aUy if it is'^W^ W thase additional interpersonal aspects 
'rSpohsibilify W syh^^ize the total the interview. All of them are worthy 
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of rmmch , inv^tlgation and "fe^» 
ba^" in the DOwse of continuing train' 
mg of mtendewfrs* ' , ^ 
1, Thm degraa of projection of per- 
sonal qualities of Interviewers into 
their jud^fienti of interviewees, 
2* TTie de^ee of influence of inter- ^ 
viewjee's pe^bnal qualities on the 
judgments of ^iAtoviewOri, 

3, Board v^sus^ coniecutive inter- 
views—the multitudinous relia- 
bility and vabdity ramifications^ 
(Oidfleld and odier British in- 
vestigators have published in this 
area/) 

4. Time allocation and expenditure 
as it effects performance of indi- 
vidual interviewerSi (My staff and 
myself did some 'work along this 

I should like to conclude my paper 
by quoting from tiiat gem of a bookj 
01dfield*s FsycMogy of th& Inimvimv 
(6, pp,a38, 139) ^ 

"If ints^iewing is an^art, the lines 
^^ng which improvement may be 
^^^ght can best be imaging by con- 
--^sidering those by which other arts have 
been gov^ned* In musiCi painting and 
the drama we find in varying degrees 
the gradual growth of a body of agreed, 
communi^ble technique arid method* 
Additions and , improvements occur, in 
part by the accutnulation of experience , 
in part by the - absorption of knowledge 
gained in other fields, and sometimes 
by deliberate effort. There are timfcs 
when preoccupation with technique 
threatens true achievement, ^ere are 
others when the Stream of traditional 
method runs^ thin and weak. The occa- 
sion may, even aris6 when^ for certain 
purposes^' the practice of an art is 
abandoned and its functions ^ Are taken 
• over by mechanical devices— when the 
e ^amophone is substituted for musical 
execution^ pi^rial representation is 
. pven over to, the camera, and the film 



deputises for the drama. These are all 
possibiliti^ inherent in the develop- 
ment of the art of interviewing. But for 
the present, as a special branch of the 
, general art of conducting human rela- 
tions, it lacks ^ but the l^eginnings of 
an a^f edi explicit technique. As this 
begins to emerge, the results may be 
surprising, ^n.d in its formation a scien- 
tifle psyehojbgy will have a large part 
to play." 
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Hie Interview as an Evaluation Technique 



NE VITT SANFORD 



THE INTERVIEW IN PERSONALITY APPRAISAL 



Even wmuN fli© felaUvely Untfited area 
o£ perionality appraisal the TOnduet of 
th© interview and the handling of the 
data it yields will depend upon the 
aims pursued and conditidns urideF 
which th© work is undertaken, He 
present paper is concemed with the use 
of the interview in a fairly comprehen- 
sive, longitudinal study of pmonality 
development in normal late adolescents 
or young adults, the interview^f being 
trained at a fairly high level in clinical 
psychology^ and the subj^ts being vol- 
unteers. My intention is to describe a 
procedure that promises to be useful 
despite the numerdui problems and 
difficulties to be anticipated, From time 
to time the confrast with what might 
better be done under different condi- 
fions will be pointed up.t 

Purposes of the Interview 

Fifteen or twenty years ago the in- 
clusion of th© interview among the 
techniques to be used in such a study 
would have been taken for. Ranted* 
Today one must seriously qu^tion 
wheth^ it is wise to use the interview 
at aU* TTi^e seems to b© general, aware- 
ness of th© filing cabinets MHout the 
cbuntTyj bulging with unapklyied inter- 
view material. Knowledge of research 
methodology has become 0 wide^read 
that the interview is often pCTceived as 
a vast tenderloin of sin arfa^errbr. The 
young investigator, trained to conceive 
of Heaven in quantitative terms, 
shrewedly induce other unsuspecting 



souls — ^teacherSp psychiafrists, employ- 
ers, supervisors and^ the like — ^to pro- 
vide the unreliable criterion measures 
while he with unassailable innocence 
provide the predictor measure. Actu- 
ally, this approach has been so success- 
ful, the objective standardized tests, 
empirically validated, have so far out- 
stripped the projective technique and 
other more global clinical devices as 
predictors of external crit^a, mg to ^ve 
rise to the hope that a new day of 
righteousness might be at hand. Still, 
there are sinners among us, those who 
believe that something has been hidden 
from our view and who exhibit an in- 
satiable curipsity about the inner work- 
ings of things. Th^ are apt to regard 
the interview as an" effective probe* 
But, today they must face the question: 
are there any scientific purposes to be 
achieved by the interview than cannot 
more efflciently b^ achieved by other 
methods? (Discussion is limited to the 
type of research undertaking outlined 
above. It seems obvious that where 
certain immediately practical aims are 
involved the interview is indispensable. 
If one has to decide whether to under- 
take extended psychotherapy with a 
given individual, or which of a numher 
of applicants to dceept as members of a j- 
small research team or to take along on 
an extended cruise, he will do weU to , 
interview them no matter what the 
battery of tests they have taken,) 

It seems safe to say, in tiie fint place, 
that in the field of personality Aere is 
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still place for Ae exploratory interview. 
In many areas the need is still to find 
the important variables, to give them 
preliminary definitions and to pin 
some notion of the signs by Svhich they 
are to be known. 

The exploratory interview also offers 
a means for seeing relations among 
vaiiables, and thus for setting up hy- 
potheses for future testing. Onfe of the 
great diftculties in perebnality research 
springs from the fact that in order to 
understand a given phenomenon it is 
necessary to take simultaneous account 
of numerous factors. It is because the 
interview, or the series of interviews, 
makes this to lome extent possible that 
it has been the major source of fruitful 
hypotheses in the field of personality. 

In the second plage, it seems at the 
present time that the interview may 
provide estimates of variables for which 
no objeetve tests are as yet available.: 
Although efficient objective instru- 
ments are produced at an increasingly 
rapid rate, it cannot be claimed that 
they have so far embraced more than 
a small section of the total sphere of 
personality variables, or that they even 
keep pace with the finding and defi- 
nition of new variables. There is a fairly 
new scale for measuring rigidity; but 
we now know that there are several 
different kinds of rigidity and of non= 
rigidity for which objective measures 
are needed. A study of personality with 
any pretentions of comprehensivene.ss 
cannot wait for the ^development of 
these tests. 

Again, if one wishes to perform case 
studies, to exemplify in detail the eon> 
mo^ patterns found in his data, to add 
to understanding of the organiza^ 
tion of personality, or to seek for new 
relationships of the kind which appear 
only when numerous variables are con- 
sidered together, he is almost bound to 
make use of interview material Even 
after the most comprehensive testing 
program, with the fullest use of pro- 



jective techniques and situatipnal t^ts^ . 
it will be found that in order to jn^ke^ 
the case "come alive" or Vhaiig to- 
gether," interview material ^^ill be 
needed. This is not merely because the 
use of the case study involves some 
commitment to an ideographic ap- 
proach, that is, to the assumption that 
in each individual the organization of^ 
variables is in some sense unique; the 
number of common traits that will enter 
the picture will Imdoubtedly exceed the 
number of available objective tests, and 
the use of available ones will involve 
more testing than is ordinarily feasible. 
More than this, the study of an indi- 
vidual persohality, properly carried out, 
will require attention to sequences and 
continuities in time; the interview is 
well designed for getting the life story, 
and in lieu of the moit comprehensive 
longitudinal study, it will have to be 
used. 

Finally, there is an area that may be 
approached either by means of the in- 
terview or by means of the question- 
naire, a decision in favor of tlie one or 
the odier depending on circumstances 
and a variety of rather subtle factors. 
The life-history, matters of fact about 
childhood, education, work and thelike 
has ^long been the special province of 
the interview. But the questionnaire has 
made enormous inroads in this area- 
and this, very probably, is why Profes= 
sor Kinsey must be at such pains to 
sliow that the questionnaire would not 
have been suitable for his purposes. 
Orxe argument for the interview is that 
certain signifieant facts of the lifc-his- ; 
tory are, for most people, sufficiently 
embarrassing or otherwise painful so 
tliat we should not expect an accurate 
account except under very favorable 
coiuliticins. I'hc unonynious question- 
naire can go quite a long way toward 
establishing tliese conditions; it is by 
no means to be scorned by those who 
desire such facts as are contained in the 
Kinsey Report, If, however, one is un- 
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d^aking a lon^tudinal studyp or any 
study of perionality that requires re- 
peated contact the same subjecti 
tfiat subject is bound to become Jaiown 
to tfie investigators* and account must 
be t^en of his sfnsibiliti^. In these 
^Qircuinstances it would be foolish to go 
on askingj by means of tfie question- 
nake, hi^ly personal questions about 
past events and cun'ent practices with- 
out obtaining infonnation about how 
tfiese questions were understood and 
reacted to. Witt^^^ respect to cstain sig- 
nificant matters we should eiqpect a 
certain amount of qonseioiis withhold- 
ing of information and a great sdeal of 
unconscious resistence or distortion. 
There is no guarantee that even the 
most careful interview will overcome 
these obstacles, but it would appear to 
be at the present time the best instru- 
ment we have for the puipose. It must 
b© emphasized that the concern here 
is with matters of faet^ with what actu- 
ally happened or what is going on now; 
where one is interested only in the sub- 
ject's tiioughts and attitudes, his con= 
ception of his childhood, his imagery of 
his parents and the like there is no ques- 
tion but that questionnaire and projec- 
tive techniques may even now go n 
long way, particularly when there ' is 
knowledgeable use of indirection. 

The Role of Theory 

Probably the main reason why so 
much^ interview material is collected 
without its ever being analyzed is be- 
cause the intervtewing was undertaken 
without thought being given to what 
was significant, and in vvhat way. We 
have commonly proceeded on the no- 
tion that if one asks about education, 
jobs, hobbies, relations of parents, ex- 
pectations for ^the future and the like 
he will get material of which something 
can be made later on. We shall prob- 
ably have to admit that what can be 
made of it in the end is directly pro- 
portional to what was put in at the be- 



ginning, in the way of theory and hy** 
podieses. The chances are that no one 
is going to theorize about one's inter- 
view material except orieself; I should 
argue that this theorizing were better 
done before the actual work of inter- 
viewing rather than after. Every ques- 
tion, or at least every line of questioft- 
ing, should have its theoretical rationale; 
one should even have thought about 
the kinds of answers likely to be elicited 
and about how these were going to be 
categorized in the analysis. It might be 
objected that diis procedure would in- 
terfere with the exploratory function of 
the interview* The answer is that, at 
this stage of our knowledge, more will 
be discovered by conceiving hypotheses 
and following them through in the in- 
terview or by, the study of material 
systematically fathered, than by float- 
ing about in a sea of undifferentiated 
verbal material. 

It might also he objected that the 
direction of interviewing and of inter- 
view analysis by theory, by conceptual 
schemes and hypotheses, opens wide 
the door through which the investi- 
gator's biases will intrude. The answer 
would be that, while bias has always 
to be contended with in interviewing 
as elsevvhere in personality research, it 
is no more of a problem in theory- 
directeJ interviewing than in any other 
kind, and that in any case bias is not 
to be controlled by the avoidance of 
hypotheses ^but rather by meeting cer- 
tain metiiodological standards in the 
conduct of the interview, and in the 
analysis of the material it elicits. 

The Conditions of the Interview 

The fact that an interview is under- 
taken with normal volunteers, the sub- 
jects being sought by the interviewer 
rather than vice versa, has immediate 
implications for thq conduct of the in- 
terview. It here becomes nec^sary to 
act in such a way as to maintain the 
subject's motivation on as high a level 
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as possible, to encourage pDsitive atti- 
tude tBwMrd the reswdi both before 
and aft^ the uitarvjew, It has, I b^ 
Ubvb, been sufflciently demonsb'ated 
tiiat a wish to take part in an inv^ti- 
gation that promise to make a contii'' 
bution to science is suffleient motivation 
for most subjectsp provided the de- 
mands upon their time are k^t wtiim 
reasonable limits. (Allowances may 
have to be made for "voIunteCT ertOTi 
however; and non-volunte^s will have 
to be offered additiDnal toducements,) 
This being so, it is ordinarily a misteJce 
to promise that the subjects "wiU get 
something out of it too,'' meaning 
WunseUing of one sort or anotiier, (But 
.this does not mean that counselling may 
not in fact have to be given, a matter 
^ to be gone info later.) 

The necessity for maintaining posi- 
tive attitudes toward the research calls 
int0^ seriQus question Ae use of non- 
directive interviewing techniques.' Since 
the subject has not sought the inter- 
view, it is of course out of the question 
for th6 interviewer to^ signify that he is 
leiHy to listen while the subject tells 
"what s the trouble''^ or "what's on his 
mind." And even though the subject 
may be brought to accept the task of 
telling about Wmself in his own way, 
we mty expect difficulty and resistence 
to mount as time goes on. The situation 
is weti-calculated to arouse anxiety and 
hostihty even in Ae healthiest individ- 
uaL Intelligent, educated subjects will, 
to be sure, go to work on some such 



eitiier diat of die subject or that of the 
researcher, And this is assuming that 
some way has been found to analyse 
tiia material. 

Non-directive interviewing, or that 
Idnd of direct^ interviewing Aat be- 
gins with the most innocuous topic and 
proceeds warily In the direction of the 
touchy ones, usually assiunes tiiat the 
subject wiU'be seen a number of times, 
perhaps over an extended period. But 
in a longitudinal study the need is for 
an appraisal of the pmonality now — 
this month or this fall— at tiie time tiiat 
otiier tests or diagnostie procedures are 
being administered. Althou^ several 
intendews may be eonsiflared to belong 
to the **now," one must be careful not 
to mistake "op better acquaintance" for 
an actual change in the personality, The 
need is for reliable estimates of the 
present state of affairs, something witii 
which later states of affairs may be 
compared, Let us accept this require- 
ment then, and ask how we might pro- 
ceed when the task is a fairly compre- 
hensive appraisal of the personality 
now, when not more than two or three 
interviews can be gi^en and these 
within the space of a few days, and 
where there is the necessity of main- 
taining sufficiently good relations with 
the subject so diat he will come again 
six months, or a year, hence' and after 
that, Interviewing under these condi- 
tions requires a schedule based upon 
theory, it seems better carried forward 
according to soma proceduras rather 



qu^tion as "What influences have had than others, and it must yield^ material 
most to do \dth your becoming the kind that can be handled in an objective or 



of parson you areP^, but an interviewer 
should have very good reasons for im- 
posing so difflcult a task— and, of 
coursa, some axfraordinarily efflcacious 
way of exploiting die material he col- 



quantitative fashion. 

The Technique of the Interview 

Let it be assumed, then, that tiie in- 
terviewer begins his conversations with 



leets. There is no denying that much his subject with a well'Concaiyed mter- 
ean be learned by pennitting a subject 
ta bring up whatever topi^ ha likes, in 
whatever ordsf he lik^, but it is diffl- 
eult to justify ^e investment of time. 



view schedule on his desk 
also ba assumad that he 
cover the ^ound 
Covering 



Allotted, 



AifB let it 
intends to 
within the time 
the ground here 
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tneaiis getting enough cnat^al bearing 
on a given topic io tiiat judges, inda^ 
pandently exiunining that mafarial later 
on, can make an appraisal of the varia- 
ble which were supposedly bfought 
into focuj by tiie ^discussion of that 
topic. 'Hie interwew schedule ib not, 
howev^, a list of the questions which 
wiU scrupulously be put to every sub- 
ject. If this were the ease, one might as 
wall have used a questionnaire in the 
first place. No, the soHedule will be 
mada up of the questions which the in- 
terviewer is asking himself; ideaUy, the 
area in question has bean sufflciently 
oifferanHated in advance^ well enough 
mapped out conceptually, so that he 
Joiows just what he wants to find out, 
The matter of how to draw thp subject 
out on the points in qu^tion wiU have 
to be left to the interviewers judgment, 
It is just herej of course, that art or 
skill or experience m^II enter the" pic- 
ture. TTie task is to get the subject to 
go far enough mto the significant matter 
at hand, without arousing too much 
amriety or hostihty or causing him to 
regret it later* ,TTiis calls for some 
ability to be aware of the subject's 
feelings, considerable knowledge of 
how anxiety and hostility are escpressed 
and, particularly perhaps, the ability to 
judge what the tafflc will bear. If one 
knows about thase^ matters he can per- 
sist in his purpose d^pita some discom- 
forture on the subjectti part,i|^cause 
ha knows that this discpmf|Kre is 
within reasonable limits and tiiS^fte can 
dispel it later on. By and large we 
should expect from the lass skillful or 
l^s experi^ced ifttej;?Hewer eirors of 
two kinds r tM^t Uircftigh over-cautious- 
ness he has failed tp bring out enough 
significant material or that he has 
elicited the material at the cost of our 
having dispmntled subjects on our 
hands, 

The experienced interviewer may de- 
mand a completely free hand in Uie 
matter of how he is going to draw out 
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the object and manage die mnotton . - 
tfiat is geuCTated, but he wiU rarely re- 
fuse to discuss witfi his colleagues Ae 
way he proposes to proceed in ap- 
proaching the particular topics of the 
intemew schedule* A result of such dis- 
cussion might very well be that a num- 
ber of suggestions about questions to 
put to the subject will be included in 
the interview, schedule. 

It should be emphasized 4at having 
accepted the general procedure being 
dScribed here the interviewer, whafl 
ever his gifts, must accept the diseiplinB 
of the interview schedule* Although he 
need not maintain any particular order 
in taking up the topics, he must hold 
himself to the task of covering them 
and not be led off in directions^ which 
might seem to him mora important in 
some particular case. The point is em- 
phasized because in other types of situ- 
ation the interviewer is more than will- 
ing to be led in directions that seem 
significant at the moment* If one has to 
make a pmctical decision based on a 
thorough-going understanding of a sub- 
ject, for example, when the question 
concerns psychotherapy ~ when and 
what kind, or when one is resf^nsible^^ 
for a tentative formulation of the cen- 
tral dynamic structure of a case, as in 
cer^in types of assessment work, the 
best procediue is quite different from 
that indicated above. In these flatter 
instances, jhe interviewer-must formu- 
late hypotheses as he go^ along, |hd 
the means for rejfcting quickly im- 
provising ones, jpend tht bulk pf his r 
tim^ followng^ up leads that point to ' 
the centrally important* Of all typ^ of 
interviewing this is the kind that calls 
for the greatest skill and knowledge, 
There will probably always be a place 
for this kind of interviewing, but the 
research setting with which we are con- 
cerned is ^not one of them, 

Contrary to what appiears to be a 
common opinion, the interview sched- 
ule is, in the research interview, fin 
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actual aid to die managetnent of anxi- 
ety and die maintenance of morale. It 
helps to lend an air^f impersonality to 
the proceedings, and it may serve as a 
convenient scape-goat, It is not that the 
interviewer wants to pry into personal"^ 
affairs; the schedule is the task-master; 
tfie intervie%ver and subject have no 
choice but to do its^bidding. With an 
interview schedule clearly in the pic- . 
ture the interviewer may put the touchy 
questions in the same even tone, in the 
same routine manner, that he employs 
with other questions; and he may, if he 
notices signs of embarrassment, relieve 
the situation by passing on to the next 
question. (In the questionnake or in-^ 
ventory that touches on affect-laden 
topics this control is lacking. An item 
may start a chain of free associations 
which leads the subject, amid mounting 
awdety, deap Into fantasy. These are 
the subjects who seek out the investi- 
gator to protest the whole proceeding.) 
In the systematic interview the subject 
may be spared this kind of painful 
rumination. Thus, there is son^thing to 
be said for the famous "rapid-fire'' ques- 
tioning of the Kinsey interview. How- 
ever much psychologist might ob- 
ject to the neglect of the psychology of 
sex in the Kinsey researches, it prob- 
ably must be admitted that the data on 
tumescence-detumescence were fairly 
accurate and collected without undue 
hardship on the subjects. 

There is, of course, a question about 
the after-effects of this kind of inter- 
viewing. I have conducted, in different 
settings, quite a large number of ratlier 
probing "one=shot" interviews witli 
volunteers and with paid subjects with- 
out becoming aware of any undue re- 
percussions, and one may well believe 
that this was true of the KLnsey investi- 
gations. The trouble is, oije may never 
know. It is my impression that we are 
safe in conducting research interviews 
of this kind provided certain precau- 
tions are taken. Obviously we must, in 



tl4 beginning, establish confidence in 
the whole investigation and particularly 
in those who do the interviewing, and 
obviously the conduct of the interview 
must be such as to keep any painful 
emotions within bounds. Beyond this, 
there are steps that may be taken to 
leave "good feeling all round" at the 
end. For example, a lapse into a non- 
directive approach, or an indication 
that the formal part of the interview 
was over and that what was now said 
was off the record, might enable the 
subject to bring out his dojibts or mis- 
givings or unresolved tensions so that 
they might.be dealt with constructively. 
Finally, the door should be left open 
for another conference at some future 
time. Although, as stated above, it is a 
poor policy to '*seir' a research inter- 
view by suggesting that the subject will^ 
get something out of it, it is wise to pro- 
vide tlic possibility of counselling, or of 
telling something of the results, for 
those subjects who make this request. 
This is not only wise, it is a minimum 
requirement of a human approach. It 
is not our wish to regard the subject's 
problems, and private thoughts and 
deeply felt sentiments as nothing more 
than grist for our statistical mill He has 
tlie right to expect human reactions to 
these human expressions. Experience 
shows, however, that few volunteers 
avail themselves of opportunities for 
counselling, and that those who do are 
nrit very demanding. 

In a longitudinal stiidy, the matter of 
counselling the research subject as- 
sumes very considerable importance. 
When a subject is seen from time to 
time over long periods some kind of 
Vclutionship between him and the in- 
vestigator, or investigators, develops 
inevitably. Thus the subject of study 
changes i}s a result of being studied. 
(This consideration juakes it all the 
more essential, in a longitudinal study, 
that all possible effort be directed to 
achieving a formulation of thfe case at 
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the time tiie study begins,) The Investi^ 
gator has no choice but to assume re- 
sponsibility for ^ese changes, and to 
act in a way that ii in die beit interesti 
of his subject A developing f elation- 
ship befrvjimi subject and investigator 
has sericms impUcations for ^cperi^ 
mental design. If for example^ one is 
studying psychological changes accom- 
panying physical ^owA, or dhanges 
resulting from education^ the effects of 
repeated contacts with tiie investigator 
may be very considerable. Such effects 
are probably minimal when techniques 
are limited to objective -tests, even 
though these be administered individ- 
ually, but the repeated *(ise of the inter- 
view would seem to be quite anotner 
matter* We cannot solve the problem 
by avoiding all counselling or by striv- 
ing to keep everything on an imper- 
sonal basis. The wis^t course, it seems 
to mCj is to make a virtue of a necessity, 
to hypothesize that subjects interviewed 
over a period of months^or years will as 
a result of that experience be somewhat 
ditf^ent in the epd froSn similar sub- 
jects who did not have the, experiencej 
to test the hypothesis through the use 
of the contol gfoupj and. to keep close 
watch upon the whole interaction proc- 
ess, ixying to observe ^ftiph aipects of 
this interpersonal p^ocj^s 'have which 
effects with which ^^^ets, 

A device that \^uld Iseem to offer ' 
several advantages fo^ research is that 
of having the sam4iiiD|fect^ Interviewed, 
within a relativei)! Haftdw space of 
time, by two or more Investigators. It is 
interesting to note ha^a f^at in the 
Tavistock Institute of Hkmjr^ ^elationsj 
which carries jOn . in flP Spirit of the 
War OflSc^ SeJeitJqp Bo^d (^e British 
"Ajsessmei^t ^C^tet^'J^ aisessrnent in 
seleo^teii Work has be^ 
just r^wp prbbeduCf- ^ 
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people in the cdurse of an afternoon), 
Thus a high premium is placed upon 
judgment, that of the assessment ex- 
perts and th^t of the executive with 
whom the successful candidate will be 
working. It would " seem that if the 
ability to judge people counts for any- 
thing in personality research— and I 
believe that it does— the interview of- 
fers the best means for enabling us to 
take advantage of it, Why not avail our- 
selves of. the judgments of the 'most 
experienced people we have, and ht 
them serve as cheeks upon one another? 
Let us divide the task of covering the 
interview schedule among several inter- 
viewers, and let them, after covering 
^ the topics assigned, offer their judg- 
ments concerning some of those per- 
vasive aspects of personality that would 
be subtly manifested as well in a dis- 
cussion of future plans as in a discus- 
sion of childhood. 

The Exploitation of Interview' Material 

The whole conception of the research 
interview, as set forth here, has been 
worked out with an eye firmly flxejd on 
the problem of how to handle the inter- 
view material. The device just men- 
tioned, that of having/several interviews 
for each subject, is an obvious attempt 
to avoid throwing out the expert'' alto- 
gether and yet to meet the minimum 
requirements of sound methodology. A 
check list of several hundred adjectives, 
to be filled out for each subject by each 
of, say, four interviewers should pro- 
vide ample opportunity for setting forth 
the uniqueness of each subject even 
while it afforded data for quantitative 
treatment. Ratings on some of the so- 
called generalized traits of personality 
can still be very usefuh as a matter of 
fact, for many common traits of person- 
ality they are still thrf best measures we 
have. Moreover, if we are going to have 
experts, and if they are going to under- 
take to penetrate the outer layers of the 
personality, then we should* have the 
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benefits of dieir Judgments concerning 
the more centoral, or the more subtle, 
aspects of the persomlity. These judg- 
nlents, too, may be r^dered by means 
of a rating scale, the interviewers having 
' agreed in advance upon the form and 
content of this device* 

But the general procedure of inter- 
viewing Uiat has bfen described here 
has been designed with particular at- 
tention to tiie objective, quantitative 
analysis of the subjects own reports. It 
is a part of the interviewer's task to re- 
cord, under appropriate headings, what 
the subject has to say about the topic in 
question. During the toterview and dur- 
ing the recording, the interviewer bears 
in mind that judges other than himself 
are going to be required to classify the 
material bearing on a given topic, and 
that the4 are going to do this without 
rafereneSko other parts of the record. 
The proposal here is that the interview 
protocol be divided, cut-up, into a num- 



ber of sections, corresponding to the 
topics of the schedule; that the produc- 
tions of all the subjects, on a given 
topic, be then assembled and handed, 
withoiit names or identifying data be- 
ing attached, to two or more judges; 
and that the Judges, working on one 
topic at a time, categorize the material 
or make eitimates of variables in ac- 
cordance with the conceptual scheme 
upon which the intey^iew was based. 
This procedure should eliminate halo 
effect and insure that Judgments on 
particular issues are not contaminated 
by" any other kinds of information about 
the subject; it should yield data thqt 
are -ready for quantitative treatment 
Measure obtained in this way, like the 
measures based on ratings by the inter- 
viewers' themselves, may be used either 
as predictors of external criteria, or as 
measures against which tests can be 
validated. 
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The Interview as an Evaluation Technique 

^ E N R Y R I C C I U T r 



SUMkARY OF DISCUSSION 



Comments on Dr, KeUy's paper 

Dr* MaQNamaFa asked wheAer it 
would be practical in terns of ttme 
and moneys to cany out Dr. Kelly s 
sugg^tiort that the eomplex task of 
the single interview^' be: fractionated 
into smaller^ units^ tach to be aisigned 
to a different sp^ialist Dr. Kelly ex- 
pressed fte opinion fliat this was > an 
empirical quisstion, in which the rela- 
tive economy of various approaches 
would need to be determined and 
evaluated. Howev^j if the only way to 
do the job properly was to use teams 
of experts, then the job would have 
to be done that way. Dr. Kelly men- 
tioned that it was interesting to note 
that he and Dr. Sanford were not too 
far apart in regard to this matter of 4 
fractionating the total interview task. 

Comments on Dr. Crissy's paper 

Dr. Speer asked Dr, Crissy to com- 
ment further on the matter of experi- 
mental designs for studying Interviewer 
consistency. Dr, Crissy pointed out 
that the ideal design for studying intra- 
rater consistency would involve keep- 
ing bot^ interviewer and interviewee 
free from any carryover effects from 
one interview to die next. For study- 
ing In^ef-rater consistency, ideally one 
would like to keep the interviewee 
uncofitaminated by successive InteK'. 
viewer effects, 

One jUuitration of tfie type of ex- 



perimental design which has been used 
in practice is seen in some recent work 
carfied out by Dr. Crissy and his 
students. These investigators recorded 
inte^vie^^ by each of several individ- 
ualSj and subsequently played back 
these recordings to a small ^oup of 
interviewers, including the original in- 
terviewer in each mw, who didn*t rec- 
ognize his own voice. When this ^Oup 
was asked to evaluate the interview 
materillp promising results were ob- 
tained in regard to both inter^judge 
agreement and consistency of the judg^ 
ment of the original intemewer. 

Another possibility would involve 
letting up machinery for routiniang 
interview research in a particular sit- 
uation over a period of years. Some of 
these ideas have been outlined by Dr. 
Crissy in a recent paper in Personnel 
Psychology. (1952, 8, 7M5) 

Dr. Peterson wondered why so ht^ 
tie attention has been given to the pos- 
sible use of classical psychophysical 
methods in dealing with interviewer. 
Judgments. Dr. Crissy indicated that 
there has been at least one rather en- 
couraging study in which "paired com- 
parison" judgments were made by su- 
pervisors in evaluating sales ^ainee^, 
each trainee being compared wift ev- 
ery other b-ainee. Such methods might 
be used profitably in situations where, 
a constant flow of interviewees U avail- 
able. ■ ' 
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Comments on Dr. Sanfor^s paper 

Dr. Syinonds wondered if there 
might be some value in having two in- 
terviews by the same int^iewer, so 
as to 'reduce the subject's resistance, 
along the lines sometimes followed in 
repeated projective testing.. Dr. San- 
ford indicated that in hii current work, 
it was hoped that resistance was re- 
solved in .the first interview, and if not, 
that it would be diisipated by tfie time 
the subject reached the second or third 
interview, Howeveri it might be desir- 
able if the same interviewer and the 
subject got together several times in 
the course of a few days. One should 
always be aware of the possibility of re- 
sistance, and one should try to deal 
with it right away. When resistance is 
observed, it would be very valuable for 
Interviewers to compare notes on it 
and to attempt to identify its causes. 

At the request of Dr, Symonds, Dr. 
Sanfo^d made a few more comments 
regarding the interviewers analysis of 
the subject's responses. In the approach 
described by Dr. Sanford, the inter- 
viewer induces the subject to talk 
about various topics, e.g. his parents, 
eta, according to the plan of the 
study. He takes notes on the materml 
/gathered in each area, dictating a more 
complete summary later. (This ap- 
proabh is preferred to that of recording 
the entire interview.) Then, independ- 
ent judges check one another's inter- 
pretation of the interview material in 
each of tlie areas covered. 
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General Comments 

Dr. Zubin made the observation that 
a most unusual sib^on seems to exist 
with regard to^tho interview— a situa- 
tion warranting some close examina- 
tloii^ Very strong criticism has been di- 
rected at the interviewing technique, 
yet most of us ^ continue to use it, to 
varying extents^ He feels that one of 
the main factors involved in this situa-. 
tion is that we tend to view the iriterT^ 
view without proper historical perspec- 
tive. Actually, the interview was among 
the earliest methods med in the physi- 
cal sciences as well ai^^in the psycho- 
. logical sciences. In first studying physi- 
cal phenomena like heat, for example, 
people were asked to report whether 
something was **hot'* or "cold." Before 
the work of Binet, a persons intelli- 
gence was often estimated by an "inter- 
viewer'' who probably judged the per- 
son "intelligent" i£ he liked him, and 
"unintelligent" if he disliked him. Some 
of the earliest studies evaluating the 
interview as a technique, yielded nega- 
tive results because they were con- 
cerned with the relation of interview 
material to entirely inappropriate vari- 
ables. 

Dr. Zubin feels tluit for studies of 
attitude and motivation, the interview 
is our best tcclinique. However, we 
need to objectify the in^erview, iden- 
tify the factors underlying it, and then 
systematically vary these factors and 
study their consequent effects* With in- 
tensive research of this sort, the inter- 
view will eventually be developed into 
a basically sound technique. 



Impact of Machines and Devices on Dei^lopments 
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THE IBM TEST SCdRING MACHINE: AN EVALUATION 



In^oduatioii 

It may SAFEjt*;.be assumed that this 
audienae needi no explanation of the 
natiire of the test iooring maehinei nor 
of tfif answer sheets on which it bites 
jrhydimically when duly stimidated, nor 
of the basic principle on which It oper- 
ates, I bnist.that it will meet, with your 
approval if I omit all tliat from my 
remarks, 

Perhaps, ^ho^ever^ *I may be per- 
mitt^ pn^ imaH; faintly nostalgie his- 
. tort^l^noj^ before I get down to work. 
At tfie ;*utset, perhaps we may remind 
ourselves'^jfliaf just about twenty years 
ago the proponents of objective testing, 
were fighting a somewhat uphill bittle 
to obtain general acceptance oj|^newep 
procedures of measurement in schools 
and colleges. As for the industrial field, 
these proeedures had as yet hardly 
created a ^ppla in that area. Except 
for a few novel d^artures, the almost 
universal Way of recording responses 
was in test booklets* Many so-called ob- 
jective tests were of the semi-objeetive 
kind which requinfd a certain amount 
of judgment on thi part bf the scorer^ 
Admittedly, these tests could be scored 
more^^Epidiy an4 with a higher degree 
ofagreement among ^aders than could 
essay examinations, but to many per- 
sons the difference was not impressive. 

At that time, in the early 1930's, the 
International Business Machines Corpo- 
ration was convtaced that mechanical 



scofing^ df lests was possible and ;wa3'^ 
actively intd^ested in producing such M 
machine. Various experiments, .w^re be- 
ing carried on in different places, and 
fearful and wonderful devices were ap- 
pearing here and there throughout tjie 
country. It was at this propitious time 
that Reynold B. JohnsoUj now head of 
the West Coast Developmental Laljo^ 
ratory ot IBM, devised the first erudf 
model pf what was to become the IByM 
. !Be$tifig. Scoring Machine. This wgs 
balefl ilot only upon the fact^ that a lead^ 
pencil mark would conduct an elecfrie. 
eurjrent; it included the much more in- 
''g*Ti|bus idea that by inserting high re- 
^'s^tprs of equal value into the circuits 
closed by the vted^ pencil marks the 
amount of current aUowed to flow 
through each of these tiny circuits could 
definitely be controlled and need not be 
afFected by the length or thickness of 
the lead pencil mark, except within 
very minute limits. 

In recent months, there have been 
persistent rumors throughout the test- 
ing fraternity concerning astonishing — » 
everi^magination-defying— new devices 
that are just over the horizon, and no 
doubt vye will hear about some of these 
this afternoon. Regardless of what the 
future may bring, however, I feel sure 
that everyone here, in fact every meas- 
urement specialjst in the countay, ac- 
knowledges that we : are greatly in- 
debted to IBM for pioneering machiiie 
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scoring, and for ,j persistent iiit^^^ 
which wili no douBt lead^tq important 
improvements, in^^^e present machine, 
I insert tfiiA brief historical note into 
my introduction because there does not 
se^ to be much, if anydiing, in print 
about tiie genesis of the scoring ma- 
dhine, notwithstanding fte importance 
of this event to tfie field of measure- 
ment, I had this spinewhat forcefully 
brou^t to my attention receptly in 
conversation with a young mail 'whose 
first and somewhat ^casudl contact . with 
^e scoring machine was during lydrid 
War II* The young man seemed slightly ^ 
maddened and disiUusioned to learn tjiat 
this hovel gadget did not spring full 
blown from the inner mysteries of the 
Pentagon Building! ' 

Some General Effects of 
Machine Spring 

. DurLng the seventeen years 6f its 
\, existence, the scoring machine 4iaSj .ofi 
bourse^ contributed enormously tp 
' l^rge-scale testing. It has done thistnot 
' \ dhly.^thrpu^h its own potential for rapid, 
inexp^hlfye scdnrigp but even more im- 
portant through , it* influjence upon ; the 
use of separate aMwer sheets. - Since 
separate answer sheet use was a re- 
quirementj if tests were to be scored 
^_ by machine, users rapidly accepted the 
separate answer sheet as a supplement 
\ to" die test booklet. This favorable 
' \climate, fycholbgically speaking, with- 
/out doubt made for more ready accept- 
ance of large-scale hdnd scoring pro- 
grams in which separate answer sheets 
werfe used. ; ■ 

I like to thiiJc that the test .^coring - 
machine has been to the testing Msi- 
nes^ what the Model-T Ford was fo the 
automobile indusfry. If the' Mpdel^ T 
put America on wheels, the test scor-'^ 
ing machine has put the youth af Amer- 
ica on objective- test answer shelts. . 

Some cynically-minded individuals 
have regarded each of these two pHe= 



nbrneria, as a not unmixed Wessing. In 
all'fairness,, it should prob.^^^ be con- 
ceded th^t the influence ^f the test 
scoring machine upon the kinds of ex- 
amination sittiations set for young peo- 
ple and the kinds of^ responses r^^uired 
may not have be^n entirely fortvi^ate. 
The use of the kind of ankwer slleetje- 
quired by the tficed/ resjptonsf-pdsitioh " 
ar^d the fixed fields -of the scoring, ma- 
chiiie has tended to force objective 
testing into a kind op stjcait jacket— in 
truth j a soniewhatv loose- fitting and: be= 
nigh strait jacket— but a s^ait jacket 
nonetheless. The four- or Bve-chpiee, 
discrete test item has become, virtually 
standard so that^ except for differences 
in content, the parts of many of our 
standard tests are rilmost as inter- 
changeable as the housing' units in a 
Levittowri. The test scoring machine is 
r^t, of course, - wholly responsible for 
this development^ But I think it has ac^ 
celerated a trend that might have been 
present regardless of mechanical means 
of; scoring/ \^ ^ ^ \ 

' Frpm the standpoint of measuremeht, 
this .,«lbse similarity in kinds of test" 
items usually as not.a seripus'limitation, 
although' it does occasiona.Uy fpist upon 
test content 4in unnatural, te^t situatiom 
For example (if I may take'aTriendlyj 
roundhouse swing j at a test of which I 
am really very fond ) , in one section of 
the Cooperative Endish Test iAllMe- 
chanics of Expressiop, an exercise in 
punctuation is forced into the multiple- 
choice IBM answer sheet form in auch 
a way that w considerable per cent of 
. riie^junior high school pupils simply do 
i.i\Qt :vunderstand what is to be done.. 
■ ini^^e is introduced what seems to be a 
kind of closure factor, which, in all 
probability, influences the resuU^r^oP 
this English test considerably, ^^Z ' - 

From a broader educational '^tapd- 
point, there may be more reason to de^ 
plore the forcing of objective test items* 
into a comparatively timall number of 
kinds in order to satisfy the require- 
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ni^nfe df .ffiy tmt ieoring machiiie, This 
proc^ure has given rise during the 
lijfc ffiCe^n years to an expression which 
is^ senfigMltti^ used by the unregenerate 
di^^hards J^OHg the critics, of objec- 
jpVB mtou^iMent as a kind of epithet 
'^"th#/ nidltiple-e^^ mind," More- 
ove^ie tiiere is reason to believe that 
therf ;:^Vmariy teachers who have ae- 
appted objective measurementj perhaps 
wiUi quiet resignationi wh© are uncom- 
fortable qyet the thou^t that regard- 
lesp of ^16 rfsourcefulilesSj s^^ imagi- 
natibri, and stimulation to thiiiking ^at 
' tfieyr bring to their teaching, in the end - 
die -achievement of their students, and 
ii^direetly their own success, is going to 
be judged largely by how well their 
stiidents respond to a single type of test 
iteih; ' , . \\ 

Fear that the^ almost exclusive usd: of 
the mult^Ie^hbica kind of test itetn 
'will have a very deleterious effect upon 
learning may have httle foimdation in 
fact; yet it does ' seem probable that 
more flexibilify in the kinds of ques- 
tion$n even to the point of intrbducin^ 
an occasional open-ended question into 
the measurement si|uationi might have 
a salutary, mflttence upon instruction 
and upon the psychology of student 
preparatioiij;; and; perhaps upoo meas- 
urement as weJL ' . 

'•■ i So. fat a^ .imeasuTernent is concernedj 
it se/ims to rhe that it has been a choice 
of valueis. In dddpting rnachine scoring , 
and in accepting the limitatiQnJi^to test 
;cons^ction imposed by tK^^ machine 
■ scoraple answer sheet, we havid.^^ferhaps 
tacitly ^ chosen to bring to the many^ 
tests not quite as good as could be pre- 
pared, under optimum -conditions for 
thevfew to whom measuWiinient might 
be restrieted in many places if; separate 
adiWer iheet procedures, were r^^ avail- 

There \^fe also the quf.^ioh of the 
extent to which the rtecessity of ad- 
justing the answer sheet to the test 
booklet introduces extraneous percdp- 
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tual and spatial factors into die meas- 
urement situation, the question of the 
lowest age' level at which separate an- 
swer., sheet teits^ may successfully be 
used; and the question of the influence 
of limitations of desk space on results 
of tests administered with separate an- 
swer sheets. There remains a need for 
more research on all tTiese qiiestions. 

But let us turn from Questions of 
these kinds to take a more down-to- 
earthj practical look at the valu^ and 
limitations of the IBM Test Scoring Ma- 
chine as- it is used in a test service or-, 
ganization, — _ — ^ 

Values and Problerfis in 
■ M:achine Scoring 

At the Educational Reeofds Bureau 
we have three test scoring machinesV 
one of which is equipped with an item 
counter, I'll comment first on the use of 
the machine in 'test scoring and then 
Very briefly on its use in item counting. 

The Bureau's experience in machine 
scoring a few million tests over a pe- 
riod of some sixtpen year^ indidates 
that, ' \Vhen thfe maohii>d is in , gdod 
working order and properly set jand 
when the an^^ver sheeCs- are marked 
with hcavys blacki glossy mai^ks with 
the reeomniended pencils and are free 
from stray dots and marks, machine 
scoring is highly efRcicnt and extremely 
^c'c^^rate, A first-rate machine Qpj&rator 
can score tests yielding a single' Scpre 
on ono side of an answer sheet at the ; 
rate of 500 or more an hour dnd can 
fnaintain such a speed for a period of 
at least two hours. When several swres 
are obtained through one insertion of 
each answer sheet, the number of tests, 
scored per unit of time is, of course^ 
lower but the' number of scores ob'-^ 
tained is much higher. 

. At tlie $artie ttoie^ it should be noted 
that test service organization iurfi as 
the Edricationai Records Bureau seldom 
finds it praotlc4falq to offer ,1BM ngia- 
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chine scoring lei^cai on. e^aln tots 
>^d^g scores on' numwous scales, 
gai^mfl^ly the Stohg Vocational In- 
ter^ Bfanks, b^usi^t the ertemely 
l^ge ntraib^ of insertions of tfie an- 



[ utod Un o r da r t o soma de fcyt 



'process, ^Tirougli the use of a '*hcat^" 
in til© niach!n|i* the answer sheets may 
b© driey. out iO th©y will yield corfeet 
BQQTm, aithough this procedure like\yiie 
tends to slow down the scoring process 



■ ^ %. ^ of of tfie answer sheets. It seems olmost 

irhp^sibla to pirsuade groups of Indl- 

m^m^'^ When the s.H„g 



l^n claim^ for tii© msehine. As a 
matterbf fact, in Ae vast majority of th© 

'.cases, th© scoring is wholly accurate. 
^\^ed the aporirig formula is S = R* 
and the eonditions of maehln© and^ 
answer^ sheet optimum, and the oper- 

' atpr reasonably alert, ttiera should prac- 
tically never be a seofing error. Wlien 

S = R there will oceasionally 

bt^a o0e-point error because no one has 
evek been able to teach the maehinf to ^ 
routi!| fractions to the nearest whole 
mimberi but experienc^, pgerators h^-^ 
come very adept at making fine di|- 
tinctions so that there should- not be ah 
. error more often than once in a hun- 
dred papers. I repeat tlia^jwhen all con-^ 
ditions ar© right a v^y satisfactory Job 
is done even with the present machine, 
which is virtuaUy unchah^d from the 
BfBX working model b?o^ft out by 
IBM, / 

..Frequently, however, not all condi- 
tions ate right. Ocgasionally, a mc^^ne 
will get out of order unbeknown to the 
operator and so syHematie errors will 
appear in the scores of a grom^of pa- 
pers. But this is a fair^*ininor mftcqlty, 



sine© such errors wnl, of course 
caught in checking either with andWfir 
machine or by hand. 

Another comparatively minor draw- 
back is that on a warm, humid day the 
" answer sheets ten# to collect^ enough^ 
moisture to affect the "heading slightly 
and thus to slow ^own the scoring 
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is done by machine. 

Departures from what is desi^ble ln 
answer sheet marking may take several 
forms— use of wrong pencils; light 
.marksi many stray marks, and the use 
of dMerent kinds of marks such as the 
dra^ving of a single slanting line across 
the rlsponse position. Several years ago, 
Dori^^Brethall of our staff and I mad© 
a little st^fly in whic| the answer sheets 
of a group ^f pupils on one of the Co- 
operative tests were recopied to form 
five identical sets except that a particu- 
lar kind of undesirablcmarking, such as 
-very Hgh? marks; was simulated 
tlyrbufthoiit each set. Each set was^ 
seared by machine, and the scores thus 
pbtained were correlated %vith the cor-* 
rect scores. Means and standard devia- 
tions were also computed. The paper 
was reported at *thc IBM Research 
Forum in Endicott in 194B. It rfeated 
no great stir among the assembled ex- 
perts, but I 4hink it did bring out very 
.. clearly the fact that while most kinds 
of departure fr^m desired marking are 
not of n^iiich ijnpfifftance, light, gray 
lookingi^arfe ler^to numerous and 
rather Wgc inaccuracies in machirite 
scoring* 

There are several ways in which a 
machine scoring unit can undertake to 
meet this problem of poorly marked an- 
swer sheets. In the first place, there is 
a nice question whether anything at all 
should be done about tliis other than;to 
score return the answer sheets. It 
may l#af^d that, since it takes 
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^ngm tot a subjeat to murk m mswmt of die machind! lliis work ordi- 

ihett w^ully h^\^. blade marks nanly t|Jc^ a large share 'of the time 
Am it doi^to m^k U^Uy and c$se- of tfie ve^ ptoipni »whd are train^ t6 
l^Iy, tiie ftdag up of peorly/markad ""operate ttie madunes, for we have 
answof §heete ^rfore icoring thmn oon^ never supceedid in hiring a cse^' of 

^ftrflid, ooDsd^tiow Individuali Since . them to i^y with thfir job. Ferhaps 



tte puipose *pf t^ting^ however, is 
. mmmMimmt^ not ^aract^ ^a^^g» 
and sinoe.Ae best approximations of 
relative a^ev^ent probably qxb to b^ 
obtain^ if aU answer sheete y^eld ^i^. 
reet Boot^i a proDedure for obtaining. 
^I^ect loarM from poorly'mu^ an^ 
swer sheets le^s a requirementt 

Various proo^ures are. possible. The 
pad^ of t^ts may be inspected, . and 
. ^ those that ^ntain many poorly marked 
^iswer sheets mafyi be scored and re- 
. 'Lft scored by hand, or potions of them 

- j\nay be handled in this way* Or Ae 

scoring unit may score all answer sheets 
by band and use the machine for re- 
' scorings or the answar sheets may be 
scored in one machme and rescored in 
' ahodier and ones having scoFing 
discrepancies then HanJ scored. If tfie 
machines differ in sensitivity, as in one 
' experiment at ,the Educational Testing 
Service, this procedure no doubt has 
considerable merit. Scoring machine 
operators, however, tend to take a dim 
view of this procedure, for when poorly 
marked answer sheets, particularly 
Jpiose with many stray mai"ks> are .run 

- tTirou^ the machine, the pointer has a 
^ ; tendehey to .creep, A few houriof ex- 
perience In trying to read a creeping 
pomter is^ hardly conducive to emo- 
tional stability, . ' ^ , 

Evwythingi considered, tlie least un- 
^ satisfactory procedure seems to. be to 



scan and. re-mark carefully all sets of ^by illuStratiori, 



f6r this particular Job spediflcationi.^re- 
marker of poorly marked aittwir^et^, 
which for some re&fonlias nev^ fbu^ \ 
its way into the Dictionary of Occupi^ 
tional Titles--^p^haps what we need to 
do is to raid a mental Institution and 
h^e some employee wIQr IQ's of about 

as^tosoi ^ ^ / . 

No^thstandin^ the great advantage i 
in spefd of macbihe^scdruig ovei^ b^d,, 
scoring when no scannmg is required, 
I doubt that, in a situation caUing for 
much soanning and re-parking, ^be 
rate of .the total machine scormg |9» 
ess "will equal the best Jiand scoring 
speed, such as LindCLuist^s workers have 
achieved at the yniyersity i^f Iowa* , 

The real cure for tbe major difficulty 
of spqring with = the present m_acli|ne is 
to strike at its source,' the marking ot 
the answer sheets bV the persons t^ing 
the test^-I think that thq^, main 'reasons 
why many answ# sheets are ^ PQO^ly 
marked are (1) that many tesl makers 
and teit publishers have ei^j^r Failed 
to' s^ess the necessity for weu maArked 
papers or they Aave buried^ these in- 
struetions= in the fine prints (2) ^at 
thousands of persons administering tests 
in our schools know nothing about, how 
machine scoring is done and ^re less; 
Mid (3) that great numbers of school 
pupils seem inpervious to speciflc di- 
rections unless the directions are re- 
iterated several times and made graphic 



answer sheets before putting them into 
the machine. This is by far pie most 
^ time-consuming and costly aspect of 
nnkchine storing, I believe our machine 
luring depar^ent spends at least 
twice as much time in scanning and re- 
v^f^arking as it does in the actual opera- 



For at least fifteen years, the Eductf- 
' tional Records Bureau has tied re- 
peatedly to persuade sc((pols to send in 
well marked answer sheets. With a few 
' notewortby exceptions, th^e attempts 
^ave met^ with little syccess. Year after 
year, ^e have sent special msbuctions 

'4 ■ ' ' . 
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'aeeemplnied by a sample weU marked, the fldminislraUon of tests, and they 
■ aniwor sheet to Jl examiners contem- may be right in their disapprpvol, It is 
plating machine sforing, but none «f what is s«meHmes ref ened ta, as a ral- 
Uiis has done much good. Not unUl the ^ulated risk. I have a feehng that the 
cateent fall pro-am. This faU we in-, test wlseness of pupils in our schools 
'^^^^Ma5€A'WTO^smttl l 4tmDv at i ont-into-aur- — and-their-knowledgejof-thejdstencam 
instruo^ntf materials. Now Ae tests oorrectlon formulas will oaus^ Uiem to. 



a 



are coming baA to ui for icoringj and 
for til© tot time in all tfiese yaars we 
saoiijo bi gettog iomewhwa* At any^ 
J* rate,^a aniwer ihaats received from a 
^ number of tihe schools are amazingly 

uMjoved* # 
p Wow the changes in special ins^cr' 
tions werq,* very sftnple. The specimen 
* a^wW sheet, ^^ch heretofore had 
s8wd to illusfrate good, black marks 
onlyi was divided into two parts. One- 
half was well marked and Uie other half 
poorly m^ked. Ipparently Ms was the 
^firsf #nf that some of our examiners 
found aut what an answer sheet too 
poprly mlh-ked to yield correct scores 
lo^ed hke. Such an illustrative sheet, 
incidentdly, riiust aQtually be marked 
by imd to get the idea across to the 
examinfr, . 

Copies reprpduced by the duplicate 
ina process simply will not do tiie job. 

The secondf change was an urgent 
request tiiat each scbool plan its test- 
ing sehedule to' allow an extra five 
minutes in which' the response posiyons 
on the answer sheets would be dark; 
ened by the pupib themselves after thfi 
test booklets w^e collected: One of my 
more realistic colleagues pointed out 
thajt, if the test were a timed one, an 
occasional pupiLmight welcome tlie ex- 
tra period as an opportunity to mark 
-responses to a .few additional items 
— more or, less at random. Naturally, I 
was 4uite_shockft# at the thought that 
any of our young people might Agage 
in such shenanigans, but we did thinjc 
tt'lidvisable to include e suggestion that 
this operation be well supervised by 
. ai^proctorsl 

I^m awartf that^^^kind of device 
is frowned upon by some authorities on 



be fairly ambivalent about wild guess- 
ing.'' 

The Graphic Item Counter 

^ There is time for only a brief com- 
ment on the item counter. This attach- 
ment to the spring machine makes 
possible a graphic count of responses 
to the various item^- as the answer 
sheets are run through the machine. On 
a single graphic item count record, it is 
pdbible to count the correct responses 
to ninety items, or to o^punt all re- 
sponses to eighteen five-dfloice items, 
and so for^. The resp^ses on 118 an- 
swer sheets may be counted fen any one 
graphic record. The graph is made widi 
carbon papes, and thus it is oot always 
as le^le as one would desire. 

'^re usi of the iteni counter unit \b 
much slower than scoring, but still it 
does count responses to items many 
times as fast as they can be counted by 
hand We have found that die item^ 
counter saves an enormous amount of 
'time in making item analyses/of new 
tests or of tests on ^ which fiurther item 
difFiculty and validity data are needed. 

An aggregate weighting unit is also 
available and is useful in research, as 
well as in different kinds of scoring, 
such as the scoring of rating scales 
where the various characteristics to be* 
rated different values, however, 

this unit is not often used, and since 
we have had virtualH^ no experience 
with it at the Educational Records 
Bureau, I will omit cojnments on that 
aspect of the scoring machine. 



Conclusion 



In summr^ry, some seventee^ years 
of experience with the IBM Test Scor- 
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fiiig MacUn® bi At Edu^ttosal Recwdf 
Bumti ^pmn indieata tiM this ma- 
iip in i^(^p m&m ia^ao- 
toiy, even wi&out furthor ^afinmmt 
Our diffleidti^ a^e mainly from fail- 
i Bff e -t&-educate t^t publish^ and^^ 



a itreamlined, iup^oniCp aleotronio, 
atomio age* For ^aniple, automatia 
feeding of die answer sheett into tiie 
machine and automatic recording of 
the s^res would noticeably increase 



mm to l^^^i^ aruoial to^r- 
• tance of wM mafked aniWiV sheetei If 
tiiis one problem can be fiUly solved, 
tiie ^dency of the vmb of fta scoring 
machine. will be in^eased at least three- 
fold. 

My rmarks have been confined to an 
tviduatipn of Ae marline m ite es^t- 
m0 forin. Althou^i as 1 have Indicated^ 
the present model is ^\ang good serv- 
ice^ so far as the..Educationii Records 
Bureau is concemedi I should like, in 
conclusion, to suggest very tentatively 
four or five kirids of improvement that 
might be made in mtt^hanieal devices 
for the scoring of tests, Without doubt, 
various ones of these items, as well as 
others, will be covered more authorita- 
tively in Mr, Bradley s talk, * 
^ First, the scoring of poorly recprded 
responseB, A machine that would accu- 
rately score answer sheets in their na- 
. tural, uninhibited, unreptoed, raw 
state as they are received from the aver- 
age student in ahnost any school or 
college would conttibute enormously to 
efficient scoring, 

Seaondi the speed of operation of the 
fnachine. The actual process of scoring 
with the IBM machine is amazingly fait 
in comparison with hand scoring. It 
does, however, depend upon several 
comparatively slow mechanical move- 
ments within the - machine, and, of 
course, it yivolves a number of move- 
ments on the part of the ^erator. 
Much seientifle progress has been madf 
even in^e few years since the machine 
was developed. It may be assumed, 
even by a non-mechanically-minded ob- 
server, that it may be possible to inb'o- 
du^ fundamental change into ma- 
chine scoring that will put these 
proc^ures thoroughly into step with 



time, reduce Ae possibility of error* , 
. Thirds the scoring of multi-scale tests. 
Any change in the ^basic sbucturf of 
the machine that would make possible . 
the rapid scoring of multi-scale tests, 
such as the Sbrong blank, after the man^ 
ner of Hankes, would enable units 
equipped vdth tfie IBM machine to 
render somewhat more mclusive serv- 
ice. 

Fourth, prodision for greater ^ti" 
in test eor^truction. At pr^ent, 
test authors who desire to adapt their 
tests to the IBM Test Scoring Machine 
must observe cortain requirements, such 
as the arrangement of the numbep of 
items in a given part in multiples of 
fifteen or else the omission of several 
numbered items, and so forth. Itit were * 
possible to eliminate the fixed fields in 
the scoring machine and also to make 
refinements so that a single answ^ 
sheet would accommodate a larger 
number of responses, these innovations 
woud assist^ both test authors and t^t 
users. 

Fifth, the eliminatiQ^n of various 
minor annoyances in the use of the scor^ 
ing machine. These include the influ- 
ence of humidity on the scor^, pins 
fliat sometimes stick, occasional batches 
of answer sheets printed so they do not 
register perfectly, and the sticking of 
scored papers in the machine, Each one 
of these is of no great importance, .but 
they add up to lost time and frayed 
nerves, Their elimination would make , 
for faster machine scoring services with 
a smile and would help appreciably |o 
safeguard the accuracy of the scores 
obtained. 

Finally, the expense. Thus far, use of 
the jcoring machine has been limited 
largely to service centers, large school 
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syitams, larg© indiiibrial organizatibns, 
and mUitary astablishinenti. The an- 
nud ^arga for thm rental and s^ice 
of tiie maehine, while modarate, has 
b^n bayond budgeti of mo it small, 

an in^^ansive '*d^k model/' or an 
elec^oniQt Qentorally located modal de- 
jigried to render nationwida senicei 



would contribute peatly to inafeasid 
use of standardized tests, to the ieoring 
of teacher-inade tests in numerous 
schools, and to research, (Incidentally, 
either of these developmerits might put 
-Q^r^E RB^nd-Wn^ed-organi^tions^ut 
of business, but happily this dreary 
topic may be omitted because my time 
has expired!) 
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Impact of Machines Mid Devices on Developments 
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THE UNIVERSITY SERVICE BUREA^ 



The tjNwaBsrrY Sbrvioe B^eau may 
ba defined in generd ai an agency 
within a univ^ity op^ftting punched 
card equipment and making to sarviee§ 
ayaUabla to any d^artapht widiin tiie 
uniVCTsity. Thou^ iuch agancies have 
exlstrf in universititi f or :twmi^-flve 
years or so* the number of lueh prior. to 
World War 11 wai quite ssftallj iand tiie 
ptjwth and ipread of diii type of ierv= 
ice during the past seven years has beien 
remarkable, _lliere are now about 200 
wlleget and universiti^ in the counfry 
which meet this general dafihitioni in 
addition there are anodier 150 or so 
schools utilidng punched card ma- 
chines that do not in that the avail- 
abUity of the madiines for other than 
their basic mission is non-existent or 
very shaiply limited. Such installations 
are mostly in the University business^ 
offices, with some in the Regisfrars' 
officii 

Among the institutions which meet 
the above general deflniUon there are 
those in which the nnichine installation 
is administered by tiie particular de- 
partment tfiat provides the basic work- 
load for the machinfe, e.g., the Office 
of die Registrar. In these cases the serv- 
ice function is more or less iecondary to 
the primary mission; we mi^t refer to 
iuch administoariVe entities as quasi 
sarviee bweaus. On the odier handi in 
about 30 of the larger universities ma- 
chine service are eenfralized in an ad- 



ministrative unit whose primary miasion 
is the service function* ITie numb^ of 
such institutions has increased four-fold 
in the past seven years, and the frend^ is 
still upwards* Since such service bu- 
reaus are expensive And -require size- 
able workloads to justify the adminis- 
frative organization/ it is likely tfiat 
such will be resti-icfed generally to the 
larger institutions, But there is no doubt 
that the concept of an autonomoui bu- 
reau that provides machine services for 
the entfre university is here to stay. * 

TOiat does the cenfralized machine 
service agency have to offer to testing 
programs and research? Those in the 
testing field have long been eager cus- 
tomers of punched card equipment* 
Testing characteristically requires large 
samples with a relatively small amount 
of information per 'individual in tihe 
sample, a situation tailored to Ae 
punched card. Testing makes extensive 
use of frequency disttibutions and order 
statistics, of bivariate disfributions from 
.coded or grouped data, of second mO'- 
ments and product moments* All of 
these can be handled on the basic ac- 
counting installation— the sorter, the 
reproducer, the collator, and the ac- 
counting machine. These machines, to- 
gether with key punches, veriflers, and 
the interpreter, for many years' consti- 
tuted the facilities of tiie miiveriity 
service bureaui In very recent ymrs 
many have acquired punched card com- 
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putm» IBM's ilow ipeed 60M^ tiiB 
m^um speed 604, or Card Fton 
grammed Calculator. It© avaflability 
of th^o macJun^ enormously increaiei 
th@ potantial of &kB s&n/ioB bureau in 
tsit^pf d'eissteg and "^aarehr^rejr^l^ 
low Qie direct i^mputation of firit and 
se^nd moments and product-moments 
in large numbers ai well as many other 
manipulationi of die data that invQlve 
multiplieation and dlviiion, ^e CPC, 
particularly, permiti for the first time 
relatively rapid computation of the 
inverse of a correlation maMc, an oper- 
ation fundamental to many areas of 
multivariate analysis ^ Without these 
computers many areas of research are 
closed to the r^earcher because of the 
gargantuan computational task, To 
complete the machine potential of the 
prfsent-ddy university service bureau 
thdye is IBM s 101 Electronic Statistical 
i^aehine ^hich permita the high-speed 
pt^AtatiiSn of contingency tables from 
bfltegorical pr coded dataj item analy- 
ses, fedfting^ and sequence checking, 
selecti^fe j0f't|ngj and many other tasks, 
ij ii S highly versatile machine that has 
'^^iJ ftv^il^ble for 400. short a. time to 
'cbnife tptoj^heral use in feSt processing, 
No mertttou^of univer- 
sity installation wdtili' tie" complete 
without pointing out that the machine 
scoring of tests is properly a function 
of such a burqau. Test sboring is a sub- 
ject of its^i "and is treated by other 
speakers; hoiveveri it is pertinent here 
to mention tnat scoring is so often fol- 
lowed diregfly by card punching that 
it is advantageous to the entire process 
to keep all operations on the data 
within a single organization, 

The complete university service bu- 
reau incoi^orates all of the above 
equipment, and can provide all manner 
of statistical and process^g services 
within a single organization— test scor- 
ing and item analysis, the preparation 
of listings and frequency distributions, 
order statistics, the computations of 



measure statistics, and finally die in* 
volved mabix computations of multi- 
variate analysis, Desirable as this pic- 
ture is, it must be recognized that it all 
costs money and requires a large wrk*^ 
ioa dr Furfeeniiure pa^zeable^achine- 
installation requires a type of ongoing 
workload that is repetitive in nature, 
one that provides die financial support 
to absorb most of the fixed costs of op- 
erating' the installation. With this as a 
base, it is possible to schedule on top 
of the regular load the many "one-shot" 
r^earch jobs diat arise in a university. 
Since the support for a university serv- 
ice bureau must primarily be derived 
from income for services rendered, die 
users must share the costs on the basis , 
of machine and personnel time required. 
This requires cost accountogi sched- 
uling, and careful planning of research 
jobs to use available set-ups and pro- 
cedi^res. For these reasons I feel that 
the Service bureaus will less and less be 
in a position to allow the consumer to 
handle his own data proeessmg; that 
is, the trend' is toward providing ma- 
chine services for the researcher, not 
jiist the machines. 

The complete service biu'eau can 
easily require an annual budget of 
$100,000 or more; it is obvious that 
many institutions simply cannot find die 
workload or financial support for such 
an undertaking. So the kind I have de- 
scribed above will necessarily be re- 
sfricted to the larger institutions, and 
many will have to be content widi a 
basic installation of^aceounting ma- 
chinos. Much can be obne with such an 
installationi of course, rfnd the way is 
opened for ^eater use of available ma- 
chine potential by the larger service 
bureaus opening their services to the 
smaller institutions, to' supplement the 
limited capacity of a small installation 
when a sizeable job arises, and ^o pro- 
vide die more expensive computing 
services that the small institution can- 
not afford. Lfcel that the large institu- 
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tion wi A ^9 €omj^e'^i^^te.'^t^i*!'~t^ planning and board wiring. 
obUgation to pi^^^aV^L^ pi^i^^Jo^' i^^ wa firequently find that it is adr 
jre^on^ baii§» an^' ibt to force desired compu- 
tus Idnd of ihariD^pt/dM(^ti@.^j^r^ into one of th^a pre-iet pattm^ 



is defin|t^ indic^^*^^ 



ratji^r than make a sp^ial madibia 



\ i^^yp even thou^ running toa irif^t 
, be iaved thCTeby* Dimng flie past yesa 
:^ ^ ;handled about 100 rasear^ jobs of 
■yfiiipui kinds, only about 10 of which 
\^rfckild ba ^nsid^ed *larga Jobs/* For 
tiie many small Jobs the pmfianent sit- 
.u^i paid off handiomaly in lowered 
' posti and persoimel time savad. 
; .* , The machines thamselv^, of course, 
'do not make a ierviea bureau. Mora im- 
portant are the people who plan the 
:^Jobs and run the machinei, An ade- 
quate staff of machine operators is 
essentiah however, the key flgure is the 
supervisor Nvho hnhdies the communi- 
cation . between tiie consumer and thie 
machines. The need for persons with 
competency in testing procedures and 
analysis, in statistical methodology, as 
well as in machine methods. Is a hi^' 
order heedj ^nd one that is rarely me* 
except in those institutions that can^ 
afford a "centtalized machine service 
agency. ThWe il ^^at nee^ for the 
avaiLibility of sqeh 'p^^scffts in^the plan- 
ning stages of ijFoJ^b, .%vith 
partibtijar emphasiljbjiiithose aspects of 
tlie pr:Oj^Qfr Uiat Jhft#;e direct iiflpact on 
, punchingsthe 'blirds andt processing the 
data^ T%6ie;^gjj||dera^^ V 
w^olve exp^Hb)^tal de$i^;;ahd stati's- 
froal *meth6iiSlK^well as^^I^ja^hnrtte? 
of coilfictirig^ and fecorfiiig iOie;' dat^^^ . 
they inyolye ; th%- d^ign Viaf^^^^ 
sheets, and :^e ^t^^tic^l ittetH^olpgy 
of item, aftd ftst Villafcilrty ^nd Valldfft^,- 
as weir (ft the mechafi|^ of ti^fc s^Hni. 

It is my ^ferfil^g; that a ti^e tinlv^OTltj^ 
service bureau mq^t ba abJ(S ptoytAe' 
personnel who' caii / cdr^ ^ out HiiS xptd 
of statistical and r^earcbttfQi^rtaritf* ^ 
and that the effectiMen^s ^f ?a' ser^^ 
bureau depends to i^ ^nsld^B1#*fe- ' 
tent upon the integratioin or Its ^eWor^^ ; 
nel into research planning/, Marty^ tiif,e 



itudi^ are mM i^a^^So^^ 
card machine, the f^r)iyt^ 
bmmu must hBiiidXpi 
Sinoa oftOTltoes' pteittrotf 1^ 
time may far moii^i^^tiii^^A^ 
important to h^t6 ^Mf^ i^^^^ pro^; 
cedmri^ and i^tdMhf ) -^t^u^ 

/^ently plannid irid b^i^r 
cost and ttae ican W^flti^tflh^ 
duced by careful plrinlttg lot th^^^ 
Job; For aseap^j '^tHe:^ ■ ws^^ 
necessary to ! tj^i qoirpladQi)^ 

among thraa - ^i^a|(I^ ^pa ■ a hfindred or 
so cases :%0i;^dm4rfltj^^ ^ik 
lator, and ©f^a;ijf :ie^ 
such a job ii tqjp/'^lmaM 
afflciantJy 0!^' aipertfiye mabhini^.iHow- 
eyer* if flflli^tha^ sit^upC -rteqi^^ 
key punch^g, tp ^the ^final ^ccuni\i1aLtions 

' -^e tom^a^l^ .^i^t^^l7le, :Uia Job 
can be dona ;toifcrijl^ gbj/and the 

* cost is /k^t;lo^*i^^B^ 

we njamtain '^ucfi a ^et-b^ £oi thi& ,0O2A 

^ 'that wflI:iyiowjfi^pm^ piie^to 

; ables . p# ^d^l^^^ aq- 
c^muIaHng. '^gu^r^^^^ We ^ 

' maintaii^ ajpentt^ft^ tabutatbr ^et-up . 

* for p^^diidn^ irt^uji 

' gnd /aorr^pijflin^^^^g^ equiva- 
lentg; peittarien^ sefc^^p for both the 

^ tab ahfl :.lflii rSta^tj^ Machine 
alloW^ ^ the; .pifepAratidn . of ' 4 ^o-way 
freqtftenby. column 
spread jhqilEoAtailyV ■Perrrianent boards 
for maWk cpiiipu^ 602 A are 

iTiaint|i(|Eiiidi ifcto is too 

lloW/ and' jWo ^liinS^ for 

/ m^bg; .^bmwto^ teoking 
fo^ard/ L to • tranif^irig these to the 

^S^d vt^^A Jit- With thf 

p^manent s^t-ups m above we 

jhf^Ie a rafliar wida ymety of sta-, 
tisti^cal;C9m^tatiOiy on m jobs with= 
otit= ri^uinhg p0^^ for ex- 
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Uie cstBm where uimw^saiy data tran- 
ieription» racGding, wad madune proc- 
m$ii$ wuld hav© be^ avoids had 
maddne ipeddii^ been consulted in 
tb^ r^^™^"g^S^ °^ ft proi^t For the 



pmpn wiA 4© smaU job=oftan a 
paduate studait^tiiie se^ae bureau 
must provide p^onnel who can take 
time to ^nmdt with and adviie 
him, to stew him clear of the enrors of 
omisiion and commission Aat make 
eventual machine analysis awkw^d or 
difflcultj to assist him m devising data 
record forms suitable for tfie use of key 
punch operatoi^, to make sure that 
cod^ are in line witfi machftie require 
ments and limitations, and to avoid de- 
signs to whidi (tiB necessary statistical 
methods are not amenable to ordinary 
machbie procedure. For the graduate 
student the service bureau personnel 
should supplement— but not supplant 
— ^his adviser. 

' 'M^0hine Sf j^aUits with a good back- 
; .^e^imfl'M^?^?!^^^^'^^* statistics, test- 
' big. aii4*?t*fc&reh methods, are not 
readily available, nor wiU they become 
so unless ttaining pro-ams produce 
thein, Hence one of tiie functions, of the 
larger s^vice bureaus is to provide a 
program of instruction in machine 
methods to tie in with backgrounds in 
;pther fields. Courses in l?asic punched 
methods, in numerical analysis, in 
l^omp^ter iio^amming, and graduate, 
^ 'tosiitantslrips for t^ service bureau are 
^^ayi ih ^ich tiiis training program 
can be orpnizedi Courses and pro- 
grams are now in existence^ but the 
large need for such training— greatly 
firtm'eased, it should be pointed out, by 



the availability of electtonic computOT 
—has become apparent just in the past 
few years, Tlie next decade will see 
enlargement of the gengral role of the 
service bureau such as to actively inte= 



grate it into tljui academic propam of™ 
the unive^ityi z*"^ 

To summarize, I have described A© 
University Service Bureau as an auton= 
omous agency within a university 
whose primary mission is tOj provide 
machine services to all departments* 
The distinction between this and otiier 
agencies that provide machine sewi^ 
in a university ^ lies in the administtativa 
organization artd the primary mission 
foD' which that ad^fntetration is ^ 
sensible, Machjr^vi^U^^ today: 
chide not only 'th^ ' r^japne statistical 
dperations that can btf tarried out on 
punched card accounting machine, but 
the more complex operations that can 
be carried out by electronic punched 
card computeri* Machine services in- 
clude a cortsulting faiietion provided 
by personnel fr^ined i^i statistical medi-^ 
ods and research procedures as wdl as^ 
in machine ^laethods; and it includes the 
-■ function of training persons %o operate 
at this consulting leveh The cost of 
stafflng and equipping such an agency 
is highj not all institutions can afford it. 
From this stems the notion that the 
service fimction must ultimately extend 
' datside the University to other Institu- 
tions. Finally it is my personal convic- 
tion that the concept of a machine serv- 
ices agency is firmly implanted and that 
it is. now a most necessary and vital part 
of the^total research facilities of a Uni- 
versity: ' \ 
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Inipact of Machines md Devices on Developments 
, in Testing Mid Related Fields 



L E D Y A R D R T U C K E R 



J ; 



tJSE^**"ELEGTftdNIC COMPUTING MACHINES 
TESTING PROBLEMS ■ . ' : 



An initial poi|vtVl wish to make about 
the elecb'onib ooinputers ii on© with 
which many computer experts \vill 
a^ee. If these devices afe ''brains," 
they are iimple minded in the present 
itata of development* They perform 
such autonnatiG loglo as the comput^^^ 
' tlotial priories or additionp subbraa- 
tspO, rflyljipjjoationj and division rand 
^mrt^e^ ifj^ based on in^ 

^equriiliti^^in. intermediate results. Con' 
sider '4 liftple clerk who could perform 
*he cdi^putational prdcesses and could 
fqllovir ^^xplicit^ coded, step by Itep di- 
rections ineludinj conditional diree^ 
tions based on inequaliyei of niuneriQal 
results. TTiiS;;qIiBtk is ^vid^ wifli a 
slate on wbiph io wf|(^^^ also 
fliat a fila ^of i tl^e^ iiheh^ inch 
cards is avaflflbl^::^^ this cleric. Th^e 
cards are teW ,|ijifj^(eiM w that each 
one. may be reJerred" to by its code 
number. At any^ pafalcular time^ one, 
and only one, ten digit number may b© 
recorded on each card* Wbenever our 4 
clerk records a result on one of tie' 
trtrds, he erases any previoui number 
on the card. 

■ Thpre will be two types of numberSj^ 
direction codes and problem numbers. 

Sirfections to our clerk wiU be coded 
such a fashion that each of the sev- 
eral numerical and decision operations 
has a unique code number occupying 



Iff ' MEPARmG this paper concerning 
j'/ . jpossible impacts of elecb^onio ;^omput» 
y on testing problems amli practice I 
had some difflculty in framing the in^ 
troductory sectionV I wanted to inswt a 
diackkner about being expert on these 
, machines witiiout implying tfiat 1 jmi^t 
* not have anything more tq s^y jn'^the 
allptted ten minutes. PrQgr^i^jh tiie 
^ devalopmait of thwe e^^iri^c^^^^ci^^ 
is the result of masterly work of an ek- 
pandiri^ group of mathematicians, phys- 
icists #nd engine^s* Soma of this prog- 
ress has ^0ien spectacularly reported in 
the preps MS development of "giant" 
brains," j'Waving several prqblwns in 
psycholo^vind toting for which I was ' 
unsatisfied 'with our present methods 
aild ^asWars, I have approached the 
field of these "giant brains" to by to 
elicit some h^lp. I have succeeded in 
being inbtiduced to a few of these elec- 
ti'onic wiwdi'^d have the beginnings ' 
of a speaking acquaifatance with a cdur 
pie* In talkii^ today I wish to mak^^rib J 
pretemQ ajt belhg an ^exp^ on dto^ ; 
fronie computm, but . raiher to' 'ap^ : 
proadi Ae topic of possible impact 
from die view point of a psychologist 
^ter^ted in quantitative mediodplogy* 
I am going to, limit my remarks to 
gmi^l pK^ait mach^^ as ^ntrasted 
to such special purpose devices as dif- 
fermtial andysw or scoring madunes, 



J 60 



^ 152 



19S3 INVITATIONAL CONFERENCE 



fte tot two di^ti of ttie directions 

nunib«ri. Code numbOT for 
wds Kavihg ttie problem numbOT to 
be u$ed are r^rdfe in die later seetion 
^f^-fts^eotion. ntimhemi lajfet 



^e of dediion itsps, Ae Mrd code 
numbOT part of Ae diraetions oode 
be tfie bode for a card containing dfreo- 
tioni for a wnditiQnal step. Normal 
progress of Ae clerk to dk^tions ^e- 
Guted will be ^nswitively md^ cards* 
. Our Mk iB now s§t up to perform 
(ha fmetlbBS prformed by an alee- 
fronic compute* Before our clerk mn 
go to workj howeveTi a program of di- 
rations has to be devis^. This is Uie 
step where brains are r^uired. I am 
tempted to say plain, ordinary human 
brains* but this wotdd not be quite, ttue 
b^us© spegial talents are involved. 
Extensive* detailed, and painstaking 
andyiis of the general problem to be 
solved is required in t|e preparation of 
a pro^m 6t dtaectioRi* This program 
has to^ self sufflcient so that once Ae 
machiri&: ii' started it can continue 



tions as wfU at^certain oth^ automatic 
logical ope^tioos. ^ ^ 

In my analogy of a clerk and flle ot 
cards I omitted mention of two tapor- 
tant matters- Ae original recordtag of 

: ■■ - : ■ : ■■ --- - a - - - --- L«J 



ffi^B pro-am and data on the eards l^l~ 
the preparation of a r^ort of r^ults. 
Input and output aspects of preset 
machine are serious problems, Insofar 
as psychological and educational test- 
ing ii^olv© much data, simple compu= 
tationsp and cbniiderable r^iilts to be 
Storfedi the input— ^u^ut problems 
are most aggravated. This raises tfii 
problem of proper \fbalarice between 
various machine facilities so as to pro- 
vide the^Qst usable services for tot- 
ing problems, Is a large, fast, tfid ex- 
pensive computing unit justified when 
coupled witfi relatively slow inputp 
output units? Would several smalte j 
and slower computing machine, *Mclv;|^^ 
coupled to the present J^e of inptifr-^ .j; 
outpiitt'units, be more tecon^ica^. ; I ; 
have b^n pleased by mfonnatipn-f^'*^/ 
tam^ftg development of. suA siac}ijh,^ ^ 



machine. ^ startea ic can ^"""^^ v. ^^'V^"^ r/^i V 
diroimh die successive steps without- ur^er a , title of ddta proc^smg ^ 
uirou^ii . wii? 3 V- r . , . - ^w.u^cf Art nUmnHvfi .^olution m the 



humaS intervention* Much time U 
taken chacking each program, frying it 
out ^n sample problems, and checking 
results so that program errors may be 
detected and corrected. 

Once a program has been devised^ 
die electronic machine has a terrific 
speed advantage oyer our clerk and file 
of cards. It is alsb less likely to make 
random errors. These advantages along 
with reduction 'of tedious labor per^ 
formed by human beings provide the 
major assets of these machines. Prob^ 
ably the most valuable, as well as spec- 
tacular, consequehce of tiiese assets is 
tiie utilization of elecfronic computers 
for solution of problems which were 
odierwise practically insoluble bectluse 
of the extensive cfeiputations required. 
Wi^ the use of electronic mnchines, 
scientists may extend th# horizon of 
problems attempted in the direction of 
problems requiring extended computa- 



/6'J 



chihe^. Ari alternative, solution U'the^ 
corin^ction- of parallel input— output 
units; to OTie large machine, This is also 

done^ " * ^ 

Those of you who are acquainted 
with the electronic machines vdll note 
that I have nctt emphasized the si^e of 
"memory" problem, This is the size of 
the card file and relates to tiie number 
of numbers that can be maintained in 
the machine. This is also critical for ; 
many testing probleitis involving much 
data. Machine developmentSi however, 
seem to be solving this problem. 

Let us try to evaluate the impact of , 
these machines bn testing* Some work V 
has already been done, especially in 
factor analysis. Charles Wrigley and,; 
Jack Neuhaus at the University of Illi- 
nois have reported determination of 
principal eoipponents. Fredme Lord of 
Educational Testing Service Is perfomi- 
ing a maximum likelihood method fa<^ 
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tor andysfa on'tfie eoinputk at Massa- 
cbu§@tt8 Inititpte of iTtchnology. 
Ertatiiive oj^atlofii stoUar to UiOia 
enc^tmtsfed to toting haxe %ien con- 

llcdtad $tat(^ flurtau of ffia^&^s^ 
Some oomiimMQ^ has been ^veirw 
posilt^ usb^of dim madbin^ op- 
eratioDs now bdng pedom^ in test- 
ing studi^# Jt items possible that large 
tabl^ of ^tef^ndatiom mn be com- 
' put^ mor^ e^Qomioally by large el^ 
tei^ maAin^ tb^ by^pr^^t itaetb- 
ods! Whetfi^ o^m anklysis operations 
can be jperfom^ more economically 
must be determined hy^ further study. 

It is my opinion that ieleetronic com- 
puters mU have meir main impact on 
t&tingij-.in term pf .pe\y procedure* 
Some of these proeedares are now auto- 
matically discarded, otiiers are npt even 
conceiv^. We mays now extend the 



PR0BLE!4^« ^ ' 

hori^n bl ,dur concepyons, New, and 
more ^tasiti limiti on feasible an^ysti 
now exists We may' now use ttwreti- 
Qally prefeited methods which have 
bgyi ^ggdri as too, pdmpl^ and at^ 



tempt solution to problems now left 
either only partially solved or lu^ol^ed^ 
entire^^r Jbxample, selection of itmk 
pairip^^lfefii, ,and larger sets. wHsh 
wiD 5M^^^^aJ£ta correlrt^n!.- 
svith solilf^^n ^terioh involves ^ph 
extensive t&hiptitations that apptoki- 
inatl^m#|y^£ifif€ now used, John Keats 
ph8 ;DaV^i?iurideri of the Educational 
T^fihg *efcs^^, are investigating tiie 
f^asibihty Iffe-^plicit solutions on an 
elecbokie &mputer» \ ^ 

As a general summary,', it is ipy opin- 
ion that the impact of electonic com- 
puters on t^tin| is a^ matter for the 
future, It will depend" on' the fertilify 
o| our imaginations. v 'Mi/ 
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Impact of Madunes and Devices on Developmei 



HAft^Y h; HARMAN* and 
B E R T H A u HARPER 



AGO MACHINES ^ TEST ANALYSIS 



Db. Tvoter has sjjokeri to you about 
eleettonia hi^ ipeed computers and 
suggrt^ A© .Mmt to which ttiay 
mi^t be adaplrf to ttie ipaoial naeds 
of toting in field of piyaholQgy. Qf 
die machine in The Adjutant G^eral i 
Offlo©, Dep^rtobent of A© Army, ^at I 
wfll diieuis dus afternoon, one Is in th© 
family of th©. el©ettonie computeri, A 
second gfoUp of machines, while 
unique, tovolves inor© eonvantiqnal 
pundhed card medipds. Aldiough my 
topic wfli cov^ heterogeneous types 
of machines designed for quite distinct 
l^objems^ nev^heless, Aey comprise 
a iingl© entity widi respect to problems 
of test analysis. 

For tta sgjk© of our present discus- 
iiDn, th^e ^Qbleins mi^t be put in 
two classes, namely (1) study of bat- 
tles of tests for the determination of 
basic psychological factors, and (2) 
study of individual tests via item stud- 
ies fbr A© development and improve- 
ment of such t^ts. The former type of 
problwn often leads to extensive analy- 
sis employing^ mediods of factor analy- 
gto. W^© W©\do not hav© especially 
powerfal fadlik^ for Ae rfeduetion of 
^ a matix of ccrrelations among many 
trtta to an a^uate factor matix, we 
db hav© spec^ facilities for the formu- 
]Mon of psySiolo^cal hypodieses re- 
sulting from factor analysit* 
. dm ipedal paaAine, known as the 



Factor Mafrix Rotator, is designed to , 
fadlitate th© work of going from an 
initial factor soluti^ to a desfrable ftnal 
factor solution (frequently described as 
^'simple structure'*) /It may b© of in- 
terest to some of you that th© madiin^ 
Dr. Tucker discussed are of tfi© digital 
variety th© AGO machin© is of A© 
analog type, that is, di© readinp ar© 
in die nature of displacements~B9ng a 
scde so that die flgures have to be 
estimated from calibraUons on die scale 
rather than read as precise digit values. 

The initial factor. wel^ts are set into 
die machine by means of a s^ei of 
dials; then die positions of di© pomts 
representing the tests ar© vi^ed as 
points of light on a scope equivalent 
to a IT'' television screen. At di© be- 
ginning of th© work, die^dials repre- 
senting the transformation matrix we 
set for th© identity ^ansfo^ation so 
diat die researcher can first view the 
plot of the points as diey appeal Ln.the 
initial (mathematical) factpr solution. 
The axes are rotated by a simple ma- 
nipulation of a diali^when desirable 
positions of die axes are loeat^, the 
researcher can^ cause the app^priate 
elements in the traniformation mafrix^ 
to be reset to take the new positions' 
into accounts 

# Since Oetobaf, 1953 associated with 
Rand CorporaUon, Santa Monica, CaU- 
fomia. * .r 



TESTING PROBLEMS 



ISS 



, Of course, tbe plofe of polnfe viewed 
on iQOpe. are two-dimensional 
spapf, that da, plots artf e^ibited for a 
pa^ of at a time* An immediate 
advanttgf dy^ hand methoda ariief 
7 f f oBo *"tte vfact Aat^: whfl©' s de^sio 
\bein^ made regardhg Ae rot aUon of a 
'particular pair of ax^, it is possible to 
vit\y)eaGh of the remaioiifig faotd^ axasj** 
in turn, ^di of ^e tWo und@r coA- 
Sidmtibn, Thm usual prootdure em- 
ployed by r^eafchOT using diis Ma- 
chine Is to \riew the plots in relation to 
aQ poASbl^ pAirs of ' ax^ mnd'Orf a 
idiematio chart to note those planes in 
which rotation is most d^irabla and * 
those Uiat mi^t be consl^^d^ order 
to keep in .number of rotations at a 
iftinflnutti. ; 

When final decisions have been made 
about the location of factor axes to ex- 
hibit simple structure, the elements of 
both the tansfonnation mfttrix and the 
final factor matr^ can be^read but as 
fast as the enperfiSienter can turn the 
dial and regid a scfilej |ince the compu- 
tations are done elecfronically. 

Factor Matrix Rotator is de- 
figned primarily to h^^ile problems of 
up tb : SO tests and 1^ factors and in- 
volving 'orthogonal ro^tioris, Thi ma- 
chin^ has already been applied' to 
problems involving up .to 130 tests and 
24 f actors . Further j it has been em- 
ployed in .conneGtibn with pbliqyue ro- 
tetions* Howeveri applications beyond 
the basic capacity of the Rotatgr neces- 
sarily involve extensive addition^ com- 
putations off the itiachine, — 

The primary advan^ge of the ma- 
dune is ^at qt spee^, sometimes as 
much as 50 to\l in ^tiQinparison with 
hand and desk c^lcularor methods. As 
a result of this speed, other advantages ^ 
accruej sych as the ability jto atihieve a 
more satisfaetory final product because 
rotations and examinations of them can 
•b^^n^^ out with iuch ease* ; 
t As indicated ear j||erj this paper is 
concerned with ^o ^yes of probleins, 



The second, povefing seveiftl machinesj 
, involves item^nalysis of tests* 

The source machine for our item 
! analysis work is tlie Document-Jo-Card ^ 
Punch* This device ecvisists of several 
jOTrente=flfstrT't5st^"5fiffg ina- 
chine chassis containing a ieniihg unit^ 
and pli^ board* The IBM answer sheet 
is fed intd the hopper of this ma^diine. 
Tlien a tabufating card^ wi^i puncHfed 
holes corresponding to flb item re- 
sponses, is prepared, by^the second 
CQmponeatj which resembles a modifled 
IBM reproducer, ^ Unique, identifying 
information for each answer sheet is 
transcribed into the punched clrd oon- 
currently by means of the third, cqm- 
ponertt, a manually, operated kejn&oard* 
ThevDocumentfto-Card Punch ^is de- 
seed to accept item information in 60 
colmnnSp al^wing the^ ^0 remaining 
columns of the punched card for identi- 
fication, background^ . and criterion in-, 
formation. Of course^ many more Uian 
60 items can be put in the 60 columns* 
For a typical Army test involving four 
alternative responses per iteprii a total 
of 180 item's can be placed in the card 
afe a single pass of* the papers through 
the machinei by means of triple punch^ii- 
ing each column.-Within this standard" 
of four alternative positions, special 
checks ha<^e been built into the ma- 
chine for the detection of double re- 
sponses and omits. For tests iriVolving 
iiv^e-ehoice items, true-false it^rfiSj or 
any typo other than fbur-choice,- tie 
Document=to-Card Punch may sfill be 
used but not all the checking features 
can be applied* ■ ' 

Considering speed of operation, it 
should be recognized that the rhaehine 
is controlled by a human operator. Op- 
timum exploitation of the mrfchine 
arises from t^e reading of a maximimi 
qfc item injFormatipn in one pass of the 
arisvver streets.- Thus, recording the je- 
ironies to a ' tes^ of * 150» four-dioica > « 
items by means, of tiws machfne, we ' ; 
estim^e that the taslc'^an be iSbne iru^p^ 
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l/7th the total Ume it would , take a 
crew of kay punch operatori to do it' 
manually. 

While item information wd£u4 hot 
ordinarily be multiple punched^ w cards 
when uitog Donvantional key punches, 
ha^dng such information prepared on, 
the DocumaM-to-Card Punch does not 
at sJl disfrart from the usrfulness of the 
tabulating cards. At the Personnel Re- 
search Brandb, the multiple punched 
cards are used for tiie study of item 
difficulty and ^incrimination on such 
conventional IBM machines as tiie 
Type 101 Electa-onic Statistical Ma- 
chine, Type 6^A Calcujatdr, and Type 
407 Tabulator. ' 

Th© Docun|ent-to«Card Punch has 
been found to be a very useful machine 
for problems of item analysis in the 
AGO over the past five years, first in a 
radier crude, experimental form, and 
' during the past two years in an im- 
proved model. During the course of our 
experience with this machine, we found 
a great need for another piece of equip- 
ment—something that would provide 
an efficient means of getting a score 
. into a punched Qard -which already con- 
tains item responses. A machine was 
designed by us and built hy^ IBM which 
does precisely fhat. A stack of cards, 
pimched with item responses, is placed 
in the hopper; the key for the test^ is 
punched in a master card; then, com- 
pletely automatically,, tiie individual 
cards are fed, matched against the key 
card, and punched wilh the number of 
instances of agreement with the key. 

This special machine is known as the 
Card Scaring' Punch. It is designed 
primarily to s^upplement the Document- 
to-Card Punch, but, of course, can also 
be used to score punched cards that 
are prepared by other means. We do 
not propose this machine as a , substi- 
titute for a test -i^pring machine. Its 



efficiency in scoring arises when item 
responses of a test are aheady ^n 
punched card form. Then, of course, 
the automatic scoring is very^. fast* It ' 
operates at a speed of about 40Q cards 
per hour, with no rnore attention on the 
.part of an operator than that of placing 
Mm cards in the hopper and removing 
them after scoring. The score obtained 
for each card is punched automatically .^ 
in any one of six fields selected in ad- 
vance by merely setUng a switch, By 
replacing the key card it is possible to 
rescore the same cards, thus obtaining 
several difFerent scorfes and' recording 
these in the same cards. "Rights" and 
"wrongsj' can be obtained separately^ 
then ariy scoring iormula can be ap- 
plied by means of a pass of the cards 
through tiie 602A Calculator. The ease 
with which various scares can be ob- 
tained from a given set of item re- 
sponses opens the way for new avenues 
of research in test development in fielfls 
such as personahty measurement. 
When trial scoring can be accomplished 
at sucli low cost, the researcher can 
easily afford to experiment with varioiis 
nicthods of keying noA-cbgnitive data. 
In summary, 'the Personnel Research 
, Branch has a broad mission, that of de- 
veloping techniques to aid the Army In 
its problems of selection, classification, 
and evaluation of men. To accomplish 
the statistical analysis inherent in this 
research most expeditiously, we have 
developed and put to use certain 
unique machines. Among such are 
those discussed here: tlie Document-to- 
Card Punch and Card Sco|nng Punch, 
which are useful in large volume item- 
analysis, and the Factor Matrix Rotator, 
a special electronic device to expedite 
an important phase of factor analysis. 
All of these have already proven ex- 
tremely useful to The Adjutant Gen- 
crnrs Office in test analysis. 
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in Testing and Related Fields 



ELMER J. HANKES 



DEVEXOPMENTS IN JEST SCORING MARINES 



I WISH TO THi^NK^a sponsors of Oiis 
meeting fq^ the opporti^nity jto tell you 
abou^yestscor's scoring %iachines. We 
hayo several kinds and I would like to 
tell you what tliey do and something 
of how ^ay do it. Please bea% in mind 
that these are all existing machines. 

First, let's considesT the machine for 
scoring the Sfrong t^ocational Interest 
Inventory. Th^ is a eqmpletely auto- 



ich % device e&nomi- 



ditions make' 
call^ feasible, 

This machine is approximately 4 fe^t ^ 
wide, 8 feejt long, and 4 feet .high. It 
has 300 vilSuum tubes, "over 'lOO.OOO 
soldered conne^on^s and requires 3 
kilowatts of electricity. It also has built" 
in checking sysffems but fr^uent accu- 
racy checks are npde.^dng an ana- 
logue device, aqeuracy' is limited but^^^ 



ma tie ^e vice— the operator inserts ai#^ the error is usually only one to <wo 
answfer sheet whicK is scanned by a stand^d points. 



resistive type picR up, This pjek up 
transfers the pattern of responses to a 
battery of 1,200 double throw switches 
whi^h are in turn connected to a ^id 
of resistances. This grid is a fix^ mem- 
ory containing all the scqringf weigbis 
' for the Strong Inventory, 

The 1,200 J witches indicate "se- 
lected" or * reffected" for each of the 
inventory questions and pu^ voltage on* 
the resis^nce grid. Current prdportidnal 
to the student s score is fed from the 
grid to a group of 50 Electronic Alge- 
braic Analysers. The Analysers then 
operate a printing mechanism which 
plots the profile of the student's inter- 
ests, ^ ^1". ^.v 

Tlie time required to score a t^t is 
tfiirty seconds. It has been computed 
that diis machine is equivalent to over 
70 of conventional type scoring ma- 
chine for doing its particular Job and 
is an Ulusbration of .the value pf tije 
special purpose machine where coh- 



J mentiolfied that a' resistance tyge 
pick up was used— we have found Uiat 
*the deperidencAyUpon the ^student for 
making a usafcfe*^ conductive pB^cil line 
is extremely unsatisfacftoty, TTherefore, 
we have s^ved^ this problem by Eg- 
marking all Strong tests with a cohdue- 
tive ink made dj^a colloidial suspension 
of graphite in water and markefed by 
A heson Colloids Company undlr the 
tradu name of Aqi^dag, 

Our experience with the^bove and 
knowledge of U*e problems inherent in 
the resistive picl^ up lead to the de- 
velopment of a very simple and reliable 
optical pick up which was incorporated 
into our next majar machine, We calh 
tliis one TUSAC for Testsoor Universal 
Scor^f and Computer. TUSAOi^ really 
designed to score test batteries, rather 
than single tests, It is too big and com- 
plex to do simple tasks. Its economic 
worth lies in its5 ability to take an an- 
swer sheet, optically scan hoih s^ea 
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simultaneoualy, gather the s^reSy coim- 
ting %boA righte and wrongs, convert 
thes^ scor^j weight artd combine the 
convef^ons and print out die resultant 
index Icores.^Jfcgle or multiple answer 
sleets can l^^used^ — ^all the clerical 
i^wgdc is done "in the machine. Fatigue 
^and resultant errois are eliminated, 
fe, TUSAC is a ^tal counter and com^ ' 
P pletely accurat^ # 
' study madfr^^t a kige milit^y in- 
ftSlation showed that a single TUSAC 
! machiql^anning ten hours a day with 
two o^^tors could replace a staff of 
100 men^ actually required for an 
, emrfg#ie>yituation. Under nonnal op- 
erations, ifl? addition to the man povvet 
^ savir^s, the savings ii^ paper alone was 
" . estimated at $14,000 a year with con^ 
siderable.. savings in space needed for 
storage|df records, Supplies, etc. 
, The TUSAC answer sheet is modellfd 
rafter the. IBM. Xhe vertical spacing of 
the response positions has been re- 
tained, byt the horizontal spacing has 
beefi doubled from Q per inch to |,per 
ingh Marking has been simplified, as. 
any kind of soft pencil o^ink can be 
used by the student. Because of the op- 
tica.^ck up there are none of th^iblank 
spaces, required by machine^ s^uctural 
considei^on and the entii^^ surface of 
the ans%Wr sheet is usa^f This gives 
over 2,100*avpilable res£\unse positions 
per side. Th^pfore, several t&ts can 
^ easily be plaffid,^" answej sheet. 
' 0At t& time I would like suggest 
tje adoption of tfMai^ard format for 
r answe^jheet^^^o that the various ma= 
chines nq^ building or under eonsider-^ 
ation vvilf DC universally us^i^L TUSAC 
ia^dSignea for jteibiUty^n this regard 
^ and sco^s both^estscor and IBM an- 
swer sh&JS, 1 

TUSAC is essentially a simple ma- 
cBine. It counts.. nwl^s on a sheet of 
paper. It counts Mem oner#t a tin^, 
just as #ou would when hand scdrin'^. 
As itfscans each mark it determines, by 
comparison widi previously aquired in- 



sfructions, whether the response is cor- 
rect or not'. If it ,is right, it adds one to 
the accumulating total— if wrong, the 
count will be affected as per previous 
instructions. A fourth or a third of a, 
point may be deducted from the sc^je 
—Simple— You ve Sdne it yourself. The 
big difference is accuracy and speed, ^ 
TUSAC is not'affected by humidity or 
cheated by stray marks or smudges. J t 
doesnt tire and .make clerical errors. 
Its scores are converted and printed 
aufeomaticflly. The output of TUSAC 
can be readily coupled to punch card 
equipment if deiired. 

The scoring rate of TUSA(^ is 100 
complete test batteries per hour at ■ 
present and sve CK^ect tp double this-^ , 
rate as we gain experience with thf 
machine, 

Dimensionally, TUSAC is not large. 
The size is only 3 feet by 6 feet by 7 ^ 
feet high. Power consumption is 4*kilo- 
watts and 800 vacuum ,tub^5 are used, 

TUSAC is a special machine an&^p 
here again we Jiave the case of a ma- 
chine being economically advisable 
only when there k a large*^ volume of 
work suitable to its abilities. ISirough 
cooperation with various test publish- 
^rs/we hope to make TL!sAC a boon to 
you through reduction of test scoring 
costs. ^ 
V In my analysis ^of the test scoring 
figld I reached the ^conclusion- tliat the 
primary consideration for scoring ^du-^ 
catioiial tests was accuracy— pepend- 
ahia acctiractj eqiiivalmt tdmhat ob- 
tained Inj wpctitive hand scoring. Next, 
^^was simplicity and reliability of the 
scorjng equipment. Repairs, . when 
necessary, should be effected by a 'type- 
writer or adding machine service man. 
Third, scores should be printed. Too 
ofto a tired operator transposes the 
sc^e, and 'reading a meter is not an 
accurate procedure. ^ 

Answer sheet arid student marking 
requirements should be simplified to 
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for tots of their own and to eliminate 
tKe need for special pencils. 

The cost of the device should be un- 
der $500 to pe^nit even imaller schools 
to take advantage of the time savings 
of test adminisbration and scoring and 
remove some of the teachers' burden. 

Por^bility would be an advantage, 
as it could ^then be used in any room 
or passed., from school to school in a 
disfafict. ^ =^ ? 

Automatic eonversioi^ to standard or 
percentile scores would be desirable as 
would be accumulation of number of 
tests scored and accumulation of totals 
for a ^pup of tests of both converted 
and raw scores. Could this be done too 
at 'a reasonable cost? 

The answer is DUS— which stands 
;for Digital Universal Scorer. Like the 
soap of U jimflar name — -DUS does 
everything. Yes, DUS does do all of the 
things outlined. Lets review them: 

Accuracy— DUS is a digital device 
with optical pick up. 

Service— DUS is made up of simple 
rugged componentSj most of which. 



ho^e stoo^ the test of time. Any type- 
. writer shoo can service it. Scores 



are 



f^ter shop can service 
printed—Raw or Converted, Uses any 
kind of answer sheet and they may be 
mimeographed from special stencili. No' 
special pencils are required. Its port- 
able—weight is under 50 pounds cased. 
Separate visual registers accumulate the 
number of tests scored and the total 
scores, 

, DUS will be available for disfribu- 
tlon in Septamber, The price will hp 
f 385 to $650 depending upon the ac- 
(^essory items dwired. 
I DUS does not do item counting. We 
are making a separate device for this 
work which will be digital in accuracy 
and havo a cupaeity of 30, 60, or* 90 
items per pass through. The basic price 
of this unit will be around $2,000 and 
it will be ready in February or March, 
That covers our work and I hope will 
cover your requirements in the test 
scoring' field. I want to thank you all 
fort^your kind attention and invite all of 
you to ,visi^,,Testscor wlien you are in 
our area^ Our staff %vill be delighted to 
see you. Thank you. 
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Impact of Machines and Devices on Developments 
in Testing and Related Fields 



F. LINDQUIST 



THE IOWA ELECTRONIC TEST PROCESSING EQUIPMENT 



TfffiRE IS NOW being installed at the 
Stote University of Iowa a new "elec- 
bronic brain" for the processing of ob- 
jective teste and test data*, *niis ^uip-, 
ment to deigned to p^onn at one time 
practically dl of die clmcal and statis- 
tical operations— scoring, tabulating, 

. computing, and raporting— involved in 
wide-scale testing programs using rela- 
tively long multiple-test batieri^, but 
it IS capable also of performing many 
other darical and statistical tasks in- 
volved in ^ucational and psychological 

^ maaiurement and r^eardi in general. 
So far ate I have been able to discover, 
the performance specifications for this 
new equipment are far in advance of 
those for any other test scoring machine 
or t^t processing equipment now in 
existence or now being developed els^ 
where. Indeed, the specifications are 
such that I believe tfiat equipment of 
this type will exfrcise a very important 
influence on the directipn of future de- 
velopments in the entire objective test- 
ing movement. My purposes in this 
paper are: (1) to tell you what this 
equipment will do, (2) to discuss the 
pq^ible significance of diis type of 
equipment for die toting movement In 
general, and (3) to explain the basis 
on which tiie services of tiiis equipment 
will be made generally available to any 
testing organimtion or agency that may 
wish to make use of them. 

Perhaps I should say, before going 



further, tiiat the installation of die basic 
equipnent in Iowa City wiU not be 
completed until almost a year from this 
datOj and that considerably more time ' 
will be required to install all of the 
presently planned extensions to tiie 
basic equipment. In certain respects, 
therefore, I may be "jumping the gun" 
a bit in making this announcemeht here 
today. Had Walter Durost not selected 
the general topic that he did for tiiis 
af^nobns prbgrami and had he not 
asked me to appear on the pro-am— 
to discuss quite a different subject— 1 
^ would surely have waited at least sev- 
eral months yet before making news of 
this project public. llowever, I felt that 
if I were to appear on this particular 
program at all, I was professionally 
obliged to make the most inter^ting 
and significant coribribution to it that I 
could— and that meant that 1 was 
obliged to tell you about the Iowa elec- 
tronic test processing equipment. 

The Iowa electronic test processing 
equipment consists essentially of a 
high-speed automatic test scoring ma- 
chine, coupled with a special-purpose 
digital elec&onic computer and a fast 
output printer. A highly flexible and 
compact answer sheet design permits 
the answers to as many as 960 midtiple- 
choice items— organised into as many 
as 14 different testf of any relative 
lengtiis— to be recorded on two sides 
of a single BW x 11'' sheet. The axam- 
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inee marks the answer sheet by sunply 
making heavy black dots in box^ oor' 
responding to the answers he thinks are 
right Any ordinary soft lead pencil 
may be used^ and no more than ordi- 
nary care in erasing is required, The 
machine will automatically detect and 
disallow double or multiple markSj so 
ttiat no preliminary scanning of the an- 
swer sheets will be needed. 

The feeding of the answer sheets 
dirou^ the machine will be completely 

V autipmatic. A stack of as many as 5000 
sheets may be placed in the machine at 
oncOi and the operator will only have 
to stand by until the sheets have been 
procossed. As the sheets pass through 
the machine* the marks will be sen^^d 
photoelectrically from both sides of the 
sheet simultaneouslyj the answers will 
be compared with the "right answers*' 
stored in a magnetic memory* and the 
scores will be recorded in digital elec- 
^onic counters. The scoring will thus 
be serial and digital in character, rather 
than of the analogue type* and will be 
extremely accurate. 

If desired, the machine will count 
the number of "rights" and "omits" 
separately for each test, and will com- 
pute a weighted composite of "rights" 
and "omits" according to standard "cor= 
rection-for-guessing" formulas. If de- 
sired, also, for a single test of not more 
than 120 items, the machine will pro- 
vide a score in which the individurd 
items are given different weights— as 

^many as 14 different weights, with inte- 
gri^ values of from 1 to 50, being per- 
missible. Agaig, if desired, the machine 
will obtain scores, either corrected or 
uncorrected, for odd and even num- 
bered itenis -^p^rately for each of as 
many as seven tests, 'so that reliability 
coelHcients may be readily cl^mputed. 
As the scoring proceeds, the machine 
will convert the raw score on "iach of 
as many as 14 tests ©h le^ch^ answer ^ 
sheet into a derived or scaled ^d6r^i 
according to a different conversion 



table for each test. Thus, the machine 
vdll convert raw scores to normalized 
T-scores, to percentile ranks, to age- or 
grade-equivalents, or to scaled scores 
of practically any type now in use. 
Furthermorej the same raw'scores may 
be converted to two or more different 
types of scaled scores simultaneously^ 
so thatj for example, the machine may 
report both ^ade-equivalent scores and 
within-grade percentile-ranks on the 
same tests at the same time. The ma- 
chine will also secure weighted totals 
and subtotals of the test scores for each 
examinee, and vvill convert these totals 
and subtotals in turn into derived scores 
comparable to those for the individuar 
testsr y 

An especially valuable feature of the 
machine is that it vvill ''read'* the exam- 
inees name and other informations 
either alphabetic or numeric, from the 
""answer sheet, It will then produce 
printed ^reports of the names of the 
examinees and all their scores, The 
output print^ywill prinV a line of 93 
characters, avfy 18 of which may be 
eitlier alphabetic or numerio* for each 
answer sheet, or, if necessary, will print 
two lines for each answe* sheet, The 
reports may be either in list form or 
in the form of individual report cards or 
profile forms for each examinee sepa- 
rately. Unles-H. very many scores are to 
be reported for each answer sheet, the 
machine will print both list and indi- 
vidual reports simultaneously* 

All necessary headings in the list re- 
ports may be printed automatically 
from blank apswer sheets on which the 
desired headings have been entered in 
place of the examinees' names, these 
heading sheets having previously been 
placed in the stack of answer sheets at 
the appropriate points. Thus the ma- 
chine works exactly like an IBM alpha- 
betic printing tabulator in this respect, 
the answer sheets taking the place of ^ 
the punched cards. 

As the answer sheets are being 

J 70 
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scored, the machine will cufnulate mch 
of the scores for groups of successive 
answer sheets, and will Dompute and 
priat the mmm of these scores at 
foot of the list report 'for eaqh ^oup. 
If desired, the machfne will also cumi^- 
late and compute the means of the^ 
squared scores' and oi M possible cross- 
products of the scores for a , liinited 
number of ' tests, "^thus providing the 
basic"^ terms needed for variances and 
for interrcoprfelation and reliability co- 
efiRcieilts. Furthermore, the machi^ 
will tabulate graphic frequency disqd- . 
butions of the soores on each test foj 
^oups of successive aoswer sheets, so 
that percentile ranks may be readily 
computed as soon as the scoring is coin-, 
pleted. Finally, 'the machine wOl pre- 
pare a graphic record of the number 
of times^ 4ach response to each item 
has been marked on a nunriber of suc- 
cessive answ^er sheets. Not only thatj 
but these grAphic item-response counts 
may be separate^.* obtained for exam^, 
inees making high and low scores 'on 
the test being scored, so that item-test 
correlations maybe readily computed. 
All of these operations— scoring, trans- 
forming, cross-fdofihg,* cumulating and 
computing of means of scores, squared 
scores and cross products, printing of, 
complete reports, ^ta^ulrtting of fre- 
quency distributions and jtem counting 
— all will be performed together during 
a single original run of the aimwer sheet 
through thfe iriachirie, ^ J* 

I shall tell you, in^ a 'riiinute tit what 
rate these operations' will l^& perjformed, 
but first let n^e say a few words about 
the scoring 'aY>d converting jjtoqass. 
Preliminary to ' scoring a group^'of an- 
swer sheets, a ''key sheet" and a "con- 
version sheet"" ni\ist be prepared. The 
key sheet consists%sentially ^a;stand- 
ard answer shBft'ori which .the right an- 
swerj have ^een marked in penci| in 
, the'ordinary fashion, and on which cer- 
iain^ ^confrbl information \$ entefe^ in 
\ ^ded formLby i^s^Ring'h^ con- 



version sheet looks just like the key 
sheet, but on it the conversion tables are 
entered in' coded form, again by mark- 
ing boxes. The key and conversion 
sheets are placed on top of the stack 
of answer sheets tq be' scored by them, 
'and' these ^are placed in ,^ the madiine 
'in the siftnf pilq witlv similar stacks for 
other test batteries "The^ machine "memi 
orizes'' ^each key and Gorkversion sfteet ; / 
a^ it comes^alotig, and' scorfs the ^uc- " 
ceeding answer sheets ^c^ordii%gly, 
■ each. key and set 'of ^donversiofi taBleff. ^ ■ 
being mitomatically erased fro^ ^le 
memor>^ as new key and €onv^s|Qn 
sheets come along. Thus, practiMly no. ; ,j 
machine time^ required for make-| 
ready or for change-Qver froin ofje 
t6 another, and tlie rnachine is capable ^ • 
of .practically epntinuous^ operation for 
long periods, even though x^%ny differ- '• 
eijt Jobs are to be handled' during any^ 



The basic rate ol tMe ^guipment Is 
6000 answer sheets per hour. However, 
the number of tests\sc6rcd or^seores ie- 
portefl is greatly in excess of this num- 
ber: Ip the ,paSe.lof 'the lowa^'^ests pf 
BasiQ Skills, fdr example, which Consti- 
tute a. 14-test^ battery with thfcS sub- 
totals, the rnachlile^Nvill obfuin, coni^ert 
an^ print ld2,Q0&:gaaarate three yigit 
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set of four profile cards for ea^^Miupil, 
with hiB iiame and scores typed across 
the top of each card, and the third is a 
, school summ^ report of the mean of 
the scor^ on, each test for eaph grade 
in tfie schooL For these purpoies we 
require a large temporafy s^aff, pro- 
vided with all of the presently avaiU 
able specializfd, equipment-such as 
electric typawriferSj comptometers, 
autor^yc computing machines^ electric 
accounting machineSj and punched 
card tabulating-equipment— that it is 
possible for them to , use efflciently. I 
cannot take time how fo describe our 
/ proeedui^es, but I%do not hesitate to 
claim that we do this work as accu- 
rately ^ quickly, and economically as the 
same kind of work is being done any- 
h where in^the country » regardless of the 
ty'te of equipment used. The staff eoni 
sis ts of about 40 unskilled workers who 
ar#?trained on the job,, arid about 20 
skilled and experienced workers* such 
^ as typists, com piiting and nccpunting 
'0' machine operators and purich^d'iicard 
tabulatihg equipment operators, *it%iies 
tftis stafi of 60 about five jyeeks. to 
handle omj program, The n^i^ ©Qiiip^ 
ment wyr^^their work in three days, 
a^ctual^ in;^iS hours of cbntinuDiis ma- 
3? chifi^^peration. I have estimated ^that 
^0 do -'the ^^e work in. thrafe days by 
preset rri^lihods would require at least 
* 400 vwarkCTf, provided^ of course, with 
■i a* oorreipondingly large amount of the 
kind of specialized equipment we are 
no# us^ng. Even this cKample, how- 
' ever^ seriously undCTstates the capacity 
of the neW ^quipn^^^inee in this ex= 
^ . ample thd equipment does not perform 
.^vmahy of tiie operations |of which it is ^ 
, dapabj^, such as^bulating fst^quer^cy 
; d^tributionS or |^m^ilatin^;^ariance 
iand' correlation data. ' ' 
- ^^Perhap^this is a good time to remind 
you that what I have been telling you 
i^ what the equipment is d^gTwd to 
doj arrd not what it has yet doh^^^aeh 
of thjl -j^tiponent operatio^sihas been 
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■ You ca^^^toiew why 1 saiip8nnay 
be "Jumping" the gi^ .a bi|J^^Mking 
thi5 announcc^c^fen 
thete are *^^^^^^^P^p^teing so 
thiui tfe§j^^W^^p:il^^^ As I 

hav^ltfeady inSiibat^^^^f^e plan- 
ning to make the .^ervlj^^rajMi equip- 
ment available at aU^^^Wt fo testing; 
and resear^ih agei^c^^enerally, =We 
hope to WQtk putji^fy of the details 
of this plan coming year,^ so ^ 

that th^pj^-;^^ vv^ien the 

mac^hin^^^^^^^^niil^ m that ,du^- 
in^ Jth^^Mi^ye will vvfsh to consult 
iliai^^ej^^^^hb are here today, so as 
to. g^^^|^'quatc consideration to all 
intere^^nnd points * of view, and we 
want^^^^give you a chance to think 
abqiut Se impliqptions of this equip- 
ment before we consult you. Another ^ 



r^^son for an early announcement is ' 
^it will give any of you who ar^ in- 
g^ed more time to get ready for an ' 
Ift^tilization of the equipment. I 
STSS^ff' guess that most test publishers 
or program directors who may wish to 
adaptjrtl^weir answer sheets ani^ related 
matertnl^lp this equipment will require 




ERIC 



164 



1953 INVITATIONAL CONFERENCE 



from 6 to 18 months for the purpose, 
not including the time needed for care- 
ful eoniidefation before taking such a 
move at alli so that for this purpose this 
advance announcement is none too 
early. 

I shall tell you more later about the 
plans we have tfius far worked out for 
making tiiis equipment generally avail- 
able. First, however. Yd like to provide 
you with a back^ound for my later 
remarks by discussing briefly the prob- 
able signiflcance of elecfronic test proc- 
essing for educational and psychological 
measurement and research in general 
These remarks will be concerned not 
just with this particular equipment, but 
with any equipment of thi^ general 
type that exploits present possibilities 
in electronics. Wlmt I shall next have 
to say, then, will be said on the assump- 
tion that the services of equipment of 
this type can somehow be made avail- 
able to anyone who needs them, 

1 don t think you will require nnich 
convincing that there is. a real and 
urgent need for the kinds of services 
that electronic equii^ment can render, 
There are now in operation in this 
counby a large number of wide-§cule 
testing programs, in each of which rela- 
tively long multiple=test batteries are 
administered to large numbers of exanv 
• inees on die same day or days or within 
a relatively short period^ time, and in 
which it is highly desiriible to report 
¥Uq results to th^ participating exam= 
inees and institutions in the shortest 
possible time. These include not only 
state, regional and national testing pro- 
grams, but also programs conducted 
within individual institutions or individ- 
ual city school systems, such as fresh= 
men placement or en&ance examination 
programs in large universities and col- 
leges, or achievement testing programs 
in large city school systems. Practically 
all of these programs involve a very 
heavy peak load of clerical work which 
can usually only be handled by a ten> 



porary staff— a staff hired, trained, and 
used for only a short time and then 
dismissed, upon each recurrence of the 
program. The recruiting, training and 
supervising of these temporary staffs 
usually constitutes the most difficult 
problem in administering a large-scale 
testing program, and in many Jnstanees 
very definite limits have had to be set 
on the program on this account^ In 
Iowa, for example, for our Basic Skills 
Testing Program for elementary schools, 
which is of twice the scope of the high 
school program I have described, it is 
utterly impossible for us to recruit in 
Iowa City as large a temporary staff as 
we need to process tlie test resiflts cen- 
trally. We have, therefore, been forced 
toi require the schools to do their own 
scoring and preparing of list report^— 
thereby placing a heavy and unwel- 
come burden on the teachers, who are 
required to do this work at no extra 
compensation. Similar situations exist, 
I am sure, in many otlifer states. With 
electronic equipment the services pro-^ 
vided to the s6liOols in this and other 
progrLims could be greatly extended, 
and the cost and udministrative incon= 
veniences to the seliools' could be 
greatly redticed. 

Not only would electronic equipment 
make possible great improvements in 
existing programs, but it should result 
in the inauguration of many new pro- 
grams as well The advantages to be 
gained through cooperative organiza- 
tion in educational testipg are extreiuely 
important, and 'Should "^be made much 
more widely available to the schools of 
the country than they are todaj. 
Through the cooperative administration 
of unifbrm batteries of tests at the same 
lime and under the same carefully con- 
trolled conditions to large numbers of 
pupils and schools, it is possible for the 
schiKjls to secure much more meaning- 
ful, more specialized and more up-to- 
date norms, to insure greater compara- 
bility in the test results from school to 
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school and frdra pupil to pupil, and to 
proflt from ttie many economic made 
possible trough largMcale operation, 
not only in t^t proe^sing but in tfie 
construction, production and di^^bu* 
tion of test materials aj well* In die 
Iowa Fall Toting Bro^am, for exam- 
ple, we are able to give Ae schools, at 
a cost of only 350 per pupil, services 
that Uiay would be unable to secure at 
saveraj tim^ the cost If each sehooF 
ware to plan and conduct its own local 
pro-am ind^endently. I a^n sure there 
are many state agencies and educational 
institutions that would now like to make 
tiia advantages of cooperative testing 
available to the schools in thto own 
states^ but that have been prevented 
from doing so primarily by the lack of 
the personnel and ecjuipment required, 
Electronic test processing, if generally 
available, should provide a new "^and 
powerful stimulus to the launehirig of 
many new testing pro-ams of a wide 
variety of types, and .should thereby 
greatly increase the scope of testing 
throughout the counhy. 

Electronic test processing could also 
have a prbfound irifluence upon the 
methods of marketing standardized test 
batteries that are primarily intended for 
independent use by individual school 
systems — particularly relatively long 
batteries such as the Metropolitan and 
Stanford Achievement Tests, the Differ- 
ential Aptitude Tests, and the Primary 
Mental Abilities Tests, Elecfronie proc- 
essing may make possible the marketing 
of such tests in much the same fashion 
as Easbnan Kodak n6# markets its 
Kodachrome film. TTiat js, the tests 
could ba sold with tnachine-scorable an- 
swer sheets on which the proeessing 
charges are paid by the consumer when 
he buys the tests, After administering 
the tests, the schools would simply send 
the 'answer sheets to a cenfral service 
agency and quickly receive back again 
individual and summary reports of the 
results at no extra pouble or cost 
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Copies of diese reporte, or of the fre- 
quency distiributions of scores, could 
be sent to the test publisher, who would 
thereby be enabled to establish and 
maintain very stable and up-to-date 
norms on the basis of die results from 
all current users of his teste, and at 
tactically no extra cost to himi At tfie 
low processing costs which electronic 
equipment an^ this method of market- 
ing should make possible, it sfeems 
likely that most schools would jimip at 
the chance to avoid ^e annoying cleri- 
cal burden which is otherwise placed 
oh their teachers and adminisfradve 
staff, and with attendant delays in Te- 
porting. Both in cooperative and in in- 
dependent testingj then, electronic test 
processing could greatly increase the 
scope and popularity of objective test- 
ing in genial. 

Electi'Onic test processing will per- 
haps be of greater int^est to many of 
you here today because of the ways in 
which it wril mak^possible new types 
of tests and tesmg services, or types 
that have heretofore been regarded as 
impracticable. Tests that 'involve very 
complicated scoring procedures, for 
example, such as weighteJ-item or 
weighted-resppnse scoring, or tests in 
which a number of different scores are 
obtained from the same items or from 
overlapping groups of items by the use 
of different scoring keys and different 
sets of weightSi have heretofore been 
generally impracticable for wide scale 
use, both because of high scoring costs 
and high costs of test development* 
Electronic processing will make diese 
and even more complicated scoring pro- 
cedures and developmental procedures 
as practicable as any others. Consider, 
for example, the possibility of providing^ 
tests, inventory blanks, attitude survey 
blanks, etc, on which the r^ort to the 
consumer consists primarily of counts 
or average scores on- individual items, 
rather than of over-all scores, What 
might be the market, for instance, for 
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ah employee inventory blank for large 
scale industrial employers, or for a 
highly ^diagnostic curriculum survey 
instrument on which percentages of 
correct responses to individual items 
are reported for all individual classes 
and schools in a large school system^? 
Such instruments and iervices cannot 
now be provided at a sufficiently low 
cost to make thena saleable, but could 
easily become practicable with the aid 
of electronic processing equipment. 

I have saved for the last what I sus- 
pect is. the most inttiguing subject of all 
for most of ^ou here today, but which 
becuuse of lack of time I shall only be^ 
^ able fo mention. That is the possible 
influence of elec&onic equipment upon 
educational and psychological research' 
in general What will It mean to you, 
for ^instance, %o be able to secure item 
analysts data, frequency distributions, 
group variances, fehability coefflcients, 
and intei*- test and inter-item correla= 
fions on experimehtEil tests as the tests 
are being scored, at only a sriiall frac- 
tion of the cost and, what is more im- 
portaht, in a very much smaller fraction 
of the time, than would now be re- 
qui^ed^ What will be the result of 
.^similar savings in time, and cost in the 
analysis of widely administered ques- 
tionnaires, personality inventories, check 
lists, opinion survey field reports, etc.? 

In milking these comments, I have 
been assuming that the advantages of 
electronic processing can somehow be 
made available to eVOTyone that can 
profit from them. How this may be 
done I can now only guess — but I do 
have sbme informed guesses to offer. 
I do nde believe it will come about 
through any widespread duplication at 
other centers of the kind of equipment 
we are now installing at Iowa, There 
. ^re two good reasons for this. One is 
that this type of equipment is too 
costly, as any of you will know who 
have given any attention to the cost 
figures for existing electronic computer 




installations. The other is that^ 
no need for more than a very 
stallations of this type. 

It is very difficult 'to estimate what 
the final cost of the Iowa equipment 
will be. We have been very fortunate 
in having had available the ^ resources 
and facilities of a large university, in- 
qluding the use of th0^. machine shops 
oi our colleges of engineering and the 
technical and consultation services of 
members of their staffj at practically no 
cost for the project, aiid the basidirloEi^ 
cal design of the fquipment has 
us nothing in cashi I think I can safely%>^ 
say, however, that had we turned over 
om-.present performance speciflcations ^ 
to any large electronic engfteering firm 
i^fi^asked it to start ttmi scratch to 
^feet these specifications, we would 
have had to spend in the- neighborhood 
of half a million dollars. 

Quite obviously, therefoi:e, it is not 
to bo expected that, this kind of equip- 
ment will be duplicated in very many 
other places in the country! It^ happens 
diat the low^ programs are large 
' enough to justify the heavy expendituri 
required on their own account, without 
counting on outside business, but, as I 
have already indicated, 'there are very 
few other test agencies that can Justify ' 
a similar expenditure on the basis of 
work already in hand. If the Iowa 
equipment is made available to them, 
even these agencies may find it diffi- 
cult to justify similar installations for 
themselves. Accordingly, until elec- 
tronic engineers find very much less 
costly ways of doing these thinp, if 
this kind of processing service is to be 
provided to the country as a whole, it 
will have to be on a highly centralized 
basis, from a very small number of in- 
stallations. 

Actually, in consideration of its 
amazing capacity, a single installation 
like that now being made'^t Iowa can; 
for a considerable time, take care of 
practically, all of the wide-scale testing 
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pro-ams in tfus country Aat are now 
able to take advantage of it, 1 have 
abeady indicted tfiat this ^uipment 
can handle tihe FaU Testing Program 
for Iowa High Schools in about ttiree 
days tima^ but it should be noted again 
tiiat this is one, of the largest of all test- 
ing programs now in regular operation. 
Most prevailing progran^ can be 
handled by tfiiS' equipment each in a 
few hours time. For Kcample, it should 
,be possible to do all the scoring . and; 
reporting for the Medical Aptitude 
Testing ^Program of ETS in a single 
dayj and yet have time for one or two 
smaller programs as well. From the 
point of view of the whole testing 
movement, therefore, it will for some 
time be diSeult to justify even one 
ad^ditional installation of this kind, un^ 
le^ If be on a purely competitive basis. 

I think I have said enough now to 
make it clear to you why we have been 
willing at the State University of Iowa 
to gamble so heavily on the develop- 
ment of tiiis equipment^ and why we 
have been so concerned about the prob- 
lem of how to utilize this equipment to 
the best interests of educational and 
psychological measurement and re- 
search in general, rather than in oiir 
own interests alone. With these general 
interestl in minds we have establishedj 
for the management of this equipment^ 
an independent non-profit corporation 
to be known as Measurement .Research 
Cencer, Iiio. Tim uriginal upsets of this . 
corporation are to be the "free time" of 
the basic equipment which I have de- 
scribed in this papeh This basic equip- 
ment is being paid for and will remain 
the property of the State University of 
Iowa. The University and the Iowa 
Testing Bcoframs will have prior claim 
on the use of the equipmenti but will 
almost certainly not require more than 
ten per cent of its time. The remaining 
90% or more is the "frejJmaL which 
Belongs to the MeasuriBment R*^earch 
Center. 



As is stated in its charter, the pur- 
poses of the Measurement Research 
Center are exclusively beneficial^ scien- 
tific and educational, and no part of, the 
net earnings of the corporation may 
inure to the benefit of any private 
shareholder or individual Infldentally, 
this is definitely not the kind of "non- 
profit" corporation in which large sal- 
aries are paid to its officers in lieu of 
profits. The principal officers of this 
corporation ai;e all full time employees 
of the State University of lowa^ and 
there is now no intention that they will 
dr^v any salaries from the corporation. 

The more specific immediate pur- 
poses of the corporation are research 
into, development of, and the beneficial 
utihzation of labor and time-saving de- 
vices in the field of educational and 
psychological < measurements The first 
claim upon the net earnings of the 
corporation will be fair research and'de- 
velopinent leading to^the improvement 
and extension of the original equip- 
ment, or to its duplication elsewhere if 
other service centers seem desirable. 
We hope in this way' to build up a 
general purposo computer laboratory 
facilitating many types of educational 
and psychological research and de- 
velopment. 

What is left of the net earnings of " 
the corporation after these immediate . 
purposes are serve^ will be devoted ex- 
clusively to ediicational research, How- 
everj it is by no means our intention to 
maximize the net earnings of the corpo- 
ration for these research purposes. On 
the contrary, our main intimate objec- 
tive^ is to encpurage the' ^owth . and 
improvement of measurement in ^gen- 
eral by reducing its cost to the con- 
sumer as much as possi We hone 
from the beginriing to offer our services 
at no more than half the cost at which 
the same services can be -obtained by 
any other present meanSj and we ex- 
pect to reduce the costs still further as 
the scope of our operations increases*. 
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You may* perhaps^ be interested" in a 
few facts about the /ievelopment and 
cons^ction of 4© equipment. The 
electrical and electronic components 
are being consfructed in Cambridge, 
Massachusetts, under a contract with 
the Educational Research Coloration, 
and imder the direct general super- 
vision of Professor Philip J. Rulon of 
Harvard Univereity. Fortunately for us, 
ERG is small enough to be able Jo by- 
pass the usually expensive and time- 
consuming red-tape of larger organiza- 
tions, and y©t is able to devote to the 
project the very best of engineering 
talent, of whom I should mention par- 
ticularly .Mr. Earl Krohnand, Mr.. 
George Hite, the senior engineers of 
ERG, Mr. Andrew Veranais, and^ Mr. 
Robert Edberg, the project engineer 
.for the State University of Iowa. THe 
mechanical components have been con- 
sbijcted or adapted by the Mast De- 
velopment Company of Davenport, 
Iowa, and in the shops of the College 
of Engineering of the State University 
of Iowa. We have been very fortunate 
also in having securfed the sympathetic 
cooperation of Remington-Rand^ Inc. 



Remington-Rand has made available to 
.us an output printfer almost ideally 
suited to our purposes, and has helped 
us iq the procurement of other vital 
equipnient also. The basic logical de- 
sign^i^-'^e equipment is my own, but 
I ai^i^ry., deeply indebted to Professor 
Rulonff^r having induced me to look 
for a thoroughgoing electronic solution 
to the problem, and for having sug- . 
gested certain general design features. ^, 
Major credit must go, of course, to om 
engineers who have succleded so well 
in translating this logical design into 
workable elecfronic terms. 

I should like tq say, in cldsing, that 
I have felt justifled in utilising my tima 
on this program as I have* because^ to 
the best of our intentions at least, the 
.Measurement Research Center is to be 
regarded as the common property and' 
in the common interests of practically 
all of you here today. Next to hoping 
that the equipment will work at alls I 
hope that it will work to the lasting 
and significant benefit of educational 
and psychological measurement in gen- 
eral. 
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act of Mac^^^d pevkes on DeYelopments 
in Te8tii% and Related Fields 
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.SPEAKING FOR INTERNATIONAL BUSINESS MACHINES 



1 A^ftociAra very mu^ die opportun- 
ity of attending tim conference and 
partioularly of appearing on this panel 
to tell you of IBM's intOf^t and activ- 
ity in the t^ scoring field. 

We have ,^eard many outstanding 
demissions and papers today on test 
sDoring machines, testing procediires, 
new approach^ to Uie t^t scorirfg 
problem, and hi^ spe^ electronic ^ 
eomputihg machine. Following so 
many fine presentations, my remarKs 
will be limited to a few statemeiits of 
fact regarding our position in the t^t 
scoring field. 

First of all let me restate a basie^^ 
tor which establishes IBM the t^&^ 
ing field. We in IBM are not testii§ 
^cpery, guidance counsellors, or psy- 
chologists. We are a manufacturer of 
tedmi<ml precision maddnes designed 
to feheve the individual of tpany man- 
ual repetitive process^ in^lv^d';%,, 
^m^utation, accounting, a^^S^i^^d^ 
keepmg. We are no mora J^ii^^^^l^;^ 
a paHem for toting proci^fe^fendi 
icoring than we are attemp&g'^^fi^it. 
the apphed scientist in dbmputation 
methods. All of you are aware, I am 
sure, of the many developments in this 
latt^ field which we have made in the 
kit few years. To name some of them " 

the 0M Electronic Cflculating 
Purieh, the Card Pro^amm^ Calcula- 
tor, tte 701 El^onic Data ftoc^ing 



Marine, ajo^ the 050 Mapietic Drum 
CaloulitpftT;^^ . ^ 

Mtoy of l^We mtfchines can be antf"^^ 
are being used in prpc^sing c^ain 
operations in the test scoring 'field. ITie 
capacities of tiieie machines are fra- 
mendoui, their ipeeds extoeniely^high, 
and their practicaL apphcat|0iii almoit 
without limit^ Jlut 'we fully realise thdt 
diese machinfeti in ^eir jpresent fornr^ 
a^ not the Answer to the majori^ of > 
your^ problems, ^ost sign^cant of the ^ 
restrictions whicK mafea them Jmprao- . ' 
Ucai ior your needs 'are, first--A^ ^/ 
tremendous expense of the machinesi 
and second— the tim§ involved in s^t- . 
ting up and pro^amming work for 
them.' . / 

Many of you are using some of our 
standard punched card units in your 
scoring, :item analysis, and evaluatioh 
operations. But we st^Jl loiow that these 
machines, at least at present, are not 
tfie ideal solution to the problems. ^ 

Wd have studied the nte^ of the 
large^ testing .agencies and ^e smaUw 
ones as well. We think we know what 
it is that you want. At least, we taow 
what you have asked for. ^ 

Before going apy further with some 
problems and ramiBcations involved in 
deigning ^a ne# ^corihr:^aehine, I\ 
would' like to i^ake an announcement. 
You may have Heard nuno^s— 1 have 
talked /s^th many of yo|i over ^e last 
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two years, "We lyive eneounto^ soma 
delays due to defanse enpneering proj- 
TOte. But— in our laboratories now, our 
©n^aers, eie wprldn^ o» a vasdy ta= 
provad t^t sfeorfig inachine. Tte spaci- 
flbations for ^ macdiine have baan 
^awn uft altar studying the problems 
you. Those of you^A whom we 
Jiayejlto iised th^e ha^^ven us your 
approval of tii«n 



iprovai 01 tn«n. ft a 

We sincerely feeUava that mis ma= 
chfaife will give ypu tfie needed results, 
Ydu wOl find that it wdl iricorporata 
those featurai which are presently laek^ 
ing in our ^dsting machine with soma 
additional ones to expedite the scoring 
Op^ation. In prin(ipla, it is similar to 
our present machine. But the improve- * 
ments which will be made will give 
/ou a far superior product and most 
assuredly you wiU find that its r^dts 
will give ypu th^ accuracy you ne^. 

The first qu^tion ydu will probably 
ask is why have we decided to taprove 
the present type machine rather than ^ 
come up with an entirely new. operat- 
ing principle, Several influencing iac- 
tors hAVe le^ided this. First— you need 
and want an improved machine witKin 
■ the foreseeable future. Although we^are 
working on new sensing and recording 
principlest they are not yet at a p^t 
where .tjipy can be made available. 
Second improved circuitry deagn 
makes possible now In^ly reli&We and 
accurate analogs gompUtihg»;,mechan^ 
isms. This is evidenced bf ^|act Uiat 
soine of the latest high 'Speed md 
large capacity computing machines in* 
iisa today are analogue machines, Third 
—it is essential that we prpducejsi ma^ 
diine which will be economically^ and 
financially acceptable fo you. I need 
not review with you the^fact thatsince 
we first intoduced a test scoring ma- 
ehtae in 1937. engineering and produc^ 
tion, costs have increased to^a pomt 
where comparisons" are staggering. 
Alost of us woidd Itice to be.t^e prpud; 
ownerf of a C&diUac but 1 dar| say that 



probabry most of us are using an autO' 
mobile which 'eosts halfes much to buy 
and operate. It' serves biir lieeds quite 
well even thojugh tiie horsepower rat^ 
ing is about half ofr the otfier and m 
general dimensions it is considerably 

.smaller, . ^ , .u- 

\ IBM is perfectly capable of building 
a machine which will meet practically 
^nyrrequirments-^fpecifijar-Howm?^^^ 



there are definitely somdTabtors which 
must be taken into consideration. Large . 
capacity inachines are very costly not 
only to build but also to, operate, Sun^ 
ilar to large production equipment, 
diese so-called "giant brains" must be 
able to support thVmselves, Can me 
testing field ai a group support tiie 
development of a machine ^which af- 
flets priiharily only the larger organiza- 
tionsP What about^the smaller dfrgamza^ 
tion currently using one or two of our 
machines, or perhaps sharing one ma- 
chiee with several others? ft is inter- 
esting to note that thase^ smaller usfrs 
' of t^lfiscoring machines constitute tiie 
■greatef maiority of the total ^numb^ 
of machines in use today, 
' Even though die argument might be 
p«^ed Oiat a few large scale m^- 
chin^; ^trategteally located, would ba^ 
able tb ^process all- machine scoring, 
tlwoughout the counby fo better ad» 
vantage, we can point out many im- 
praeticaliUes in ^is idea which more 
than -offset the advanta|es. We and 
Other organizattons have established 
large computing and accountmg cen- 
ters tju'ougho&t the counbry wi^ the 
most powerful machines known to 
handle every type of problem, .Hmv-^ 
ever, we hava hadmo indication that 
these machine centers are ^plac^ 
any numbers of smaller .Installation 
^So it is v^iiCh our test scoring inachirte,; 
Even though many of you are offering 
. test scoring services on a wide basis, 
our gato< indicate that the single ma- 
chine users list contLnues to increase. 
We are enthusiastic with you aboift'the 
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future powtii of A© 
tii^efore, b#liav@ tibat 
improvad low cost ma 

4^ large user and the imaU ii 
ttia flrit im]^rtant st^ ti^ the deyijjQp- 
Dfient prop;^. I wo^d Ucg to re^- 
phaiiza abo tiiat this pia^ma ii \^at 
you have: aake^is rnddca, ^ > . 
v. i^May I (wnpnisli© a^ Aat this ma- 
jSMog. do ^ not repr^ent our final in^ 
tar^t in this project. If it were I 
would not have reeommandad that. we 
even considar buildiBg it IBM will 
build almost any machina asked for on 
a spaoial conttaet b^s where funds 
for such a project at^ available. We 
may have discouraged some of you on 
such proj^ts ^at tim^ whan anginaer- 
ing and production facilities 'w^e not 
availabia. However, wa are interested 
in such projects for your field as well 
as others, ^ 

But our 'own research and develop- 
zn^t projects eontinua in our labora*^ 
tpries, and we knoi^ that soma of them, 
will definitely aflFeot your interests. We 
are working on new devices and prin- 
ciples l^hich will be standard ^eguip- 
mtnt in the future. A machine capable 
of functio|jing as a scoring machine 
alone and also which could be tied in 
with other standard data processing 
units is certainly ftie ideal J We belipve 
that fiituro scoring hiactA^ will be 
forthcoming in our normal documenfl 
scanning at^ti sensing, computings and 
data^ processing machirte development 
^o^am. These will be * the answera 
to ^ur many needSj both pracUcally 
and economically. 

y NoWi what are some of the improve- 
ments of our proposedr^ew machine? 

First and most nUded — answer 
sheet mark dis^imination. The ability 
of the padiine to determine those an- 
swer sheets which are not machine 
scoraable along 'with othe^ sheets in 
a ^oupi 

S©eondT--*an'^, improved scoring key, 
Tliird— improved calibration and 



stl^iili^ of circuits to elin#\gte drift 
and cbnbnuous ch#d^ng* > , y v ^ . 

Fowth— jncreased mot^ capacity 
and sQcrin^ cpinbinatioiis. - m^^- - 

Plftii-^imj^rovtid feeding and.fasfer 
Qpe^tion^,^^ / s . 

Sixth-^simplified operationi reading 
Qf^d marking of scores, ' 

Seven^— ovpray improvement in re- 
;abilit)4*JbQ£h^^|tg_Qpar£ition_tf 
macBine and#ervice* ' 

Therf^are otheri.ywhich have ^o do 
with the engineering technicalities, but 
all of these will, I assure you, lead to 
a tremendousl)4. imprwed macKine. ^ 

Wa have considered other features 
such as automatic feeding of answ^_ 
sheets and printing of icoras, as well as 
automatic programming for formula* 
scoring. Wa will be happy to consider 
these further llirith any of ^eu who feel 
yoa would like to have them. EuPfSt 
the present they will not be added as 
standard unitfh This is in the interests 
of cost to you anJ "because wa are op- 
timistio about how some of our future 
devclopments-.will be able to servs^^ou. 
We are also desiroul , of getting the 
new machine to you as soon as pds- ^ 
iSible. When laboratory hiodels are com- 
pleted they will be placed on t^st in 
the field and at that time we will ask 
you^ assistance in evaluating the. ma- 
chine. 

We atf glad to se| some oth^ new 
test scoring machines^ being inU^duced^ 
to the field. It would be most presi^p- 
tubus of us to think tKht ,we were the 
only ones capable of building a market- 
able scoring machine. With thesg new 
machines more peopf^will be working 
on and suggesting ^cfiine Aprove- 
inents and design chatoes^^^B^^rea- 
tive ideas wiU - be oirwOT*loproduce 
sqgring macHifies whidfc will better 
serve you." Afid^ natura^flyj the more ' 
different oper^ng principles Employed 
the more complete will be the cover- 
age' of the maopitie scoring field, 
' Finally, we in IBM want you to 
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Imcm that we wffl ^ntinue to be in^ 
t^^ted fn asiistitog you, W© hold our 
QMOiMUon vMi^ t^^g ifU bi 

and MB^tmoB yra have m and 
we look f or^wd ^ ^ntinuing gur fine 
rdatloiis wiii. you, , Yduf iugg^tions 
VfOl always be wd^me and we are 
opnfidoit ^th Ae doie cK^op^a 

be able to prodace for «ou tfje ina- 
diines wUeb &mwerye\3^ future 



It has be^ra a peat pleasure for me 
to be wii you today asd althou^ my 
qgqiiRfntencB witH -membe^ of tiui 
^up has been growing rapidly for 
Ae last two y^i, tfjii' meeting hw 
been of fremenfjous ^asiistanc© to me. 
i am looking forwd^d to my visits to 
your office and I hope you will con- 
Jor agiistan^ when 



problems ariie and .when you feel Aat 
we may be able td' taprove our ma» 
ohfties ^d ior^rtce to you. 
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Appendix 

PfllrtiGipant^T^1953 Invitational ^ 
Cptifarence on* Toting .Pioble^a 



Edu^tioa I « 

An^fg^ Joe K^sm^Yt B^nl P611^ge 
Amaii Domfty C Usivm^ of I^o A 

j^nM^Am^ Jant^ ^fjbiaiomd Examp 

AmiANN^ J. Stanley, CbmeU Undvaiiify 
Alijeh, Kfargartt j^rUand FubUb 

Swiob _ _ _ , ^ ■- - ■ . 
Aj^jimt^ fiD^ B. Jr,» EducAtion^ Testing 

AiMJoii John E., BotEon| Univsmty 
Anasta^ Anne., 'Fofdham Univei^ty 
ANnmALTSR, O. F., itJ Louis Ui^verdty 
AiitmiOH, Paul R.^ FeELDsylvaoia College 

for Women ( 
ANDimoNj Rosa 6., The Fsyehologica! 
' Corporation I 

Roy North Carolina Stat© 



i^-Qn " B^^oti^Tj -§^riioof WTj^John Mwto^ 
Associatei 

Bekw^ Nelson Fit. State UstmUW Teach- 
ers CoUega (N,Y,) 



AKxiR£\^t T. 0«i tJnii 



j^, of Mai>^land ' 
iduoational Teiting 



^GOFF, Wmiam H*, Educational Tesling 
. Serviea. H •^->7 . X ~ . 

^ AHTHONYi Cf* A^liaiPi jMaryland State Pe- 

pMrnent of Eduaiaon ^ J., 

Apwm^, Val^tiQe^-^-Aidi&rdion, Bellows, 

Henb^ & Cd,, Zsc. / ^ 
Agysfp^ Jos4 G,, Uniyariity of Puerto Weo 
Aj^cpTBDHQ, C3^1^ New York Stat^ 
/ Education peparbnant 

ibfiiAOKO, FpWj li^gh"^ ttniversity, - 
sroi^j^S^tfiel T.jiBrown UniVersify 
ff, SSth, Spstegfidd 
A^teAH* ftsie, Golun%ia University ^ 
Ba^^ Paul C., Pumu© Unlvwiity 
B^TUMUBj Earl %, Sdence Research 

BjMmtf c3ia;les jL tQrosby High School 

CWateAury, Coria J x 
limooc^- Hertot D., New York State 

D^t* of CJ\il Sej^j^ _ 
BAnmSy Paul J*, Wprld Book Co. ' 
BABTHn^ R. v., Equ^Uonal Testing Serv^ 

r - ' 

Ba^ot^j a. Fe^nnel Research Br. 
AGO , 

BEA^]fi[0p KaWtrine, USDA Graduate 
i^ool 



Bsi^NTi ^ro&y M., Nor&ampton S^ml, 

for Glrli _ 
BENEDWVf Gi Gi, Fhilhpi Aoadeniy 
Bekkbtt, G^rge Ki» The F^^olo|^^ 

Corpowition 
Bensd^, ^r^ur Lp, Edu^tton^ Tasting 
Saivioe 

BERDm^ Ralph F,^ Upivarsi^ of Minnaioto 
BsROEHf Bmiazd, Munidpd Civil Servlca 

" CommMon (N. Y.) 
Behoesen, B. ik, Pewnnel Inc? 
Bishop, Rtith, Natjoml League, for Nursing 
Blaessea^ Willard W., Universify of Utah 
BLANCiwDj GdnoU Nf The Signal Schooli 

Fort Monmouth 
Blaul, Rp Eiizabethi Highland Park High 

. School (lUinoii) 
; Bloom, B, S*, University of Chicago 
JBoBBiTT, Joseph M., National Initituta of 
Men^ Health _ ' 

BoLLENBActmn^ JoaUj Gin^nnati Public 
> Jchools - 

BoNNEH^ Hu^rtt. Columbia University 
BaAGA,^ Susan E,, Arghdiocasw Vocational 
S^rwioe 

Bhj^l^v, Fhihp H.» Intmattonal Businasi 

Nfachines Gorppration 
BRiO^v tij^fmn^ Amerioan OcQupadonal 
^^apy^^odation , * 

BBi^feo^X%honks L,, -New York' Civdl 
. ' ^ervi^ * \ 

Ba^t^, F. Kenneth, National Aisocmtfon * 

of Mmufactufers , 
Bray.i Daugks W., Columbia University 
Bhto^s^ Claud© Fm Worl43ook Co. 
Bwrow, WilLUun H.^ Bureau of Cu^€U» 

luhi R^^r^ , \ 

BBQimNo^ Angela M.j iDeparteient of 
Edupitioi^ (Ms^land) 



Cecitft., N, Y. Ci\il 



BrolV: 
Servil . 

Brooks'J Ridiard B-, College of WiUiana 

& Miry ^ 
BfiowK, rradaridc S., Great NJUe PubUe 

S^ooh '(N. Y.) ^ > 
Bryan, MUam M., Silver Burdett Com 

p^y ^ ' \. 
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DAMVi John T;, Buieou o|*aval Person- 

nel ' : ^\ 

! Edu 



BkYAN, Ned,, Rutgon University 
^K'-l5^ ^t?tl.«to T., Y. State Eiueatlo^ Dc 

BwuBi J^w M.. Darien S*~l« d^S? Harry WaU«r. Richardson. Bel- 

"'^^^ Sri-'m^l^'Slary. .1^ City College of 

Frgd B., Hunter CoUeg© ^ 
Da,^^J. Sanford, Reieareh Branoh, OTfR 
%J^ON, Hu^ M.* PenmylvaBia State 

DfiTCifflk, Lily* Pennsylvania State CQUege 

foS Women i ■ 

Diamond, l^rralne Krugloy, Jeachers C6U 

lage,. Columbia University 
Dm^%* Taul B.* Edu^Uonal TesUng 

Dtai* Helen A*., VocaUsjal Advisory 
Service 

Dion, Robert, California Test Bureau 
' Dosto, jQhj(iy E., Educational Testing^; 
SwRce 

Doppelt/ Jerome E.^ The Psyahologieal ' 

Corporationi i rr i ' 

DRAGOiiTz, Anna, Educational resting 
, Servi* * . , . 

Draeb L. University of Wis^nam 
Dressel, iaul L„ Midiigan State College 
Dry, Raytftond JW Insurano© Agency 
MoBagement ^idtiation 
^uBois^PhLlip ^, Washington University 

trtd Ad- 



■ land 

CAF^i Mri. John W., SouUi Cwolina State 
^Aa^HOs&ge • \ 

Carsy, Robert 1,, Board of Eduction 

(YonkefipN.Y,) - ' : ^ ^ 
QabAson, C* Ba^ond, Air Gomm^d and 

Staff Schobl, Aabama j 
Caimon, Harold 8,^ Upsala Q)Uege 
Cabrou-, John B„ Harvard University 
OXsi* Eth^i 1.. Pol5^eehflla^.Mtute of 

Brooklyn . ' , 

Gavne, Bernard S., Boston Umy^sity 
Cavhe, Helen M,, Massa^usetti institute 

oi Technology _ - - 

CiMiK, Joieph B., New York State De- 

partaient of Civil Service . - 
. CHAVNCTnr, Henty* Educational, TC^ng 

Senric© - , ^ « 

CHxmcimAi Rutii* ^tiOwi OoUege 
Cu^i Willis W., Cahfomia Test Bureau 
Cldt/ Norman, pducational Testi^ Serv^ 

Cobb, William % Pennsylv^a State Col^ 

lege ^_ w 

CociLm/ John H., Temple Univeriity 

doFFi^; Wflliari E., Educational Test- 
fog Service ^ ^ i 

C^HEN, Ailip S,, Montclai| State Te^h- 
ei^ College . ^ ^ ^ 

CoiUBMAN, WiUiorji, University of T^nes- 
sea # ^ 

^KGBR, ^ouis, N, Y. State Educatioj De- 
partment . ^ S ^ 

CoPELAND, Herman A., AtlahUc Reflning 
^Company 4 \ t S 

CoWLEs, John T., vEducatonal Testing 
Sefvi* ■ , 

Crawford, Barbara, Educational lesting 

^ Service fi^ ^ ^ n i. 
Grissy^ William L E., iQBeens Collie 
Cbonb"ch, Lea J.i University of Illinois 
Cnoo£, Frances E., World Book Company 
Cro%vley, Hmy L.^State Teachers Col- 

^ Ime (Massa^usetS) : 

^ Cuffi^flNcs, i^lana, 'Bie Psydiologieal Cor- 
poration _ • F 
dfciETON, pdward E., University of Ten- 

Ct^SoN.ft^uise W.i KnoxvUIe, Tennessee 
i Cynamok, Manuel, Bproklyn Cd^ge 

'At- '■ ' 



DuNN/ Frances E„i^rown University 
Spymp^'t, Walter "N^? Test Service an( 
^ ^ttnent Center ^ 
DYfifc Henry So College: Entrance Exam- 
ination Board FT 
Ebel, Robert L., State University of_Iowa 
JiEhgelhart, Mm D., Chicago Publie 

Scho^ ^ ^ i-, 11 ' 

Enrick, Ralph, Michigan State CoUege ^ 
Epstein, Bertram* The City College of 
New York _ , , ™ 

' Fan, Chung-Teh, Educational Testmg 
Ssrvice , _ . 

Fay Pad J„ New York State Department 

o^ Civil Service 
Feldt, Leonard S., State University^ ot 

Fels* Williani C., CoUege Entrance Exam- 
ination Board ^ ^ « i ^ 
Ferguson, William C, World Book Com- 

pany , p 

FiNpLEY, Warren C, Educational Testing 

Service . _ . * jr * 

Finkle, Robert B., Metropohtan Lite In^ 
surance Company ^ 
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ftJiiikQAH, John C., America Inititut© to HAmpaa,, A. E., Educational Testing S^rv* 

Resfarch ' . , Je© . 

TimmiQ, Mae/ Edueatlonil Tasting Sflrv' Ha^eb, Bertha P., Personnel Besearch 
ioe '\ : ' - Brandh, ^ ^\ ^- . 

f^^hWG, Edwin G., B^on pigebw \ Habt^ Wlay E.^ Babtock and 
, OrganizatiQi^ . s " i P^^t^y ; ' . ^ 

FL»ram^ Fmdc^*M., Jr., Tiba Ohio Sti^e HAnvEV, Philip B.^ Univarsit/^f Conneeti-. 

Univei^ty ^ ^ eut . ^ , 

FoL*v, John P.s Jr*, Tlie P^ehologil|^ Cor^ HA^IIfcs, Jff "^oj^as, Univershy't^f lUinoii 

poration 4 , , * ' HEAiflJ 

Fomjm^ Jeim, F,| Eduoational Testing'^ V^aryi! 



' Fo5c, WiUiwai H4 Indiana University » 
FBjimMlmi Eli* Personnel ^Riesearcn 
. Brai^, TAGO * 
/'f^M^^msEN* Nojfmani Educatipnal Test'^ 
. Ing Service 

FREEikfAK, Paul M,, Educational Testing; 

Service * 
F^iNGH* Beniamin J., New York State 
» CivU Sen^iee - ^ 
. Vb^HOHi John W.| Edu^tion^ Temng 
"Service _ ^ * *^^- 

French, Sidney J., Colgate? Uriiver 
FRffiDMAN, Sani D*, New York State 
'I pitftaent of CivH Service 
Frtoman/ Sidnayi Buleau of Navid 
ionnel V ■ 

FuRiT, Edward J.| University of Michigan ^ 
diU^LAGmR* Henrietta L,* Educational 
^4 Testing Service , 
. .^ABDNER, EriQF.i Syracuse University ^ 
..Oabhison^ Harry W., Educational Testing J 
% -^^rvice _ ' / = ■ 4 

^^©AWKOiWj Roman, World Book Company 
t^EOssi* Nonnan* Temple Univeriity 
T^XJNK^ Marjorie, The Psycholpglpal Cor- 
'^J, poraHon „ 
9 @^B^iCH, J*^ Baymond, ynlver^i 
' Ck»ngecticut 
^ Glass? Albert A., Fo; 
GRBfibB* Edward B.* 
GmsENE, Hairy A,, 
I Iowa 

Gheeb* George D., Hi 

search Office 
ORBkm, Elaine Pri 
tion Service 



TtgATONi Kennj 
. ^nry and- ' 
HroftMAN!^, 

BeUows* fienry 
HEL^cKji John 
Service ' > 
Heroni^^us, a* 
*Iowa; 

HlTnNGER^ 

, \ CoUega 0 ^ ^ 

v^HosfcRMAN, Solomon^ Civil 
mission (New ^ork ) ^ |: 
IqrnbosteLj Victor O,* Nai 
Assodatiori 
BQWK^i Milton W:* Quae: 
' ^, Clark W.j Dartmi 
ESTONj Edlthj 



K -f ii . 
for j^syehglogical 

chi^dsoh. Bellows, 



vtion^ 




ehna, George Wa; 



dod " tukn^Ju^ersity: . , 
i^^vi^ve P.i Arcnfi9<tesa%,Voj^p^^{ 
larviee ' '^i- Sl^Ski 

jBferfW. B.^^Ontari^^llege o|^^ 

' al '^^co 




\^ ^ ja^^to^^'^Educational^^^ 
. J A^H^^S^apj ; National Lec\gue^for. Nurt- ' ^r&r 

N-j:'^— ^ \- " 

.E>bration JtofFtfev^ W^ W^i^^afnar^^ " ^ 

ersi^ of . J^lwiNok; Mel^ H:^ Brookiyh^CoUuga.;^:!^^^* ^ : 
■ i ' J^HN^,; A; P^^Education^/^^ 



Resoifiees Re- .u^ice!. *^ . . . (iij 

jpHNiO^^ ^Lewis W,, personhal pejpari- 

ment^SCi^ prPkiladd^ 
JpHEs* Uaxmj W^'* ^ffltajT-Aeadeipy ; 

GulLdejin, Harofdl .E3ucaUosai,^Tcaftng*^ New Ybrje i ■ ' A i A i^^^ 

Service " ^f- /. ^ ,Kali% Roberti Educational Tesflhg^€erv- 



GRO'^^s, Kenneth J.* 
Did, 

Gustad, John W*,'Um^r^^ ^j^^^ryland 



Hacih, Elizabeth, Te^'^^C^^;^flQ^■ Karpi^ ftrt, EduQ^Uonal'pitfijg Serviee 
lumbia University ^^^^^f J^i^ ^^ 



HAGcraTYi Helen, Persbfigtf BeieaiSi^^ KEL^rRdbert J.j University ^^hutejota 

tioni.TAgB^ _ ■ .^'^ / -^^^^ DeCourey, Edpcatidh&l ^.Tesflng 

Conn/)%^'I^ ^^"^ Paul, ^Mfcatiofaafc ,Tepting 

New^d3^ Sta^ Depart- A \'Servlcej^ - . 1/ 

Service ^ - ? IteLEY^' ^uman '1^*, Frpfessor Emenhisy 

jTastsco^ Harvs 

Bi, Mfltptf, Massachusetts KellYj 




^ University - 't*^ 

Jewell, University of Mi^gaM 
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r,^, John P., Whtte PWns* New York 



t ■ wKfiEte, John W.rMlAigan Stole CbUeg© 

J Wiffl aai E*i the CSi^ta fc^pql 

Besmirch ^ 
iCoLnmoCp F*, Edu^^nal Testing 




KomscSt Mm State Taa^era i^Uege 
. ( Boston, Mass.) , 
IQjns^ G* Frederic, Didco TMvwity 
EimHNEa» Rose :£i^ Gi^ CbUege bl New 

York >. v \ ' ■ ■ 

Kvraffii^ Qiari^^j jtoertcw r Fubli^ 

H^Cb A^ocda^n 
XvAi^^Vif ; ' ^1 ^ Bc^qn ^ Univfiriity 
. Schobl or Edudatiori . 1 . * 
. luiOBi^T, Iowa Stela Tea^igra GoU^g© 
LAiK^cupii d. R., Syracuse ynivemitt^ 
' ^Ali&iQtJii . G, V. j EducaMonal Testing 

Si^vi^^• . . 

i4Yt^H, Wilbitf Uniye^ty pi Minna- 
'-^iiMfta* -■- * - - ^ - - " ■ -''^ - 

' itoo,^ Eliz4bf4i Aiharican puBMe Health 
' ^. Asidadon - ! ' 

Vl^^d^i Kent W,| Univ^filty of Mifchigan 
I^^^iUf, . jtaW M./ Li^ldand Air Foto 
= 3as# ''='jjO ''^ " ■ - ■ ^ ■* 1 .v// 
tm^o^.^ mm World Book Coinpany 
■ W^/ j£S|€©h*^^ NeiW York Stele D^fflrtment 

: of ^tf Serv^ V ^ * 
Vlmmmp ■ HolUs ,M,i American 



MddiM^. 'University Qfij6ttUi, 

MqGAXCBBroGEyWBarbafaj EdMitipnal Teit^, ' . 
/ ing ^ryiee ~ ^ ' \ V 

MgCanW,. For^s E,J Pi^pnel ^^e^rt- • 

nghti City of Madelplfl** v ^ ' • 
Maw^^ Joyce Jtl New York State De- * 
- «T ^^^^^^kt Civil Se^ce ^ ^ ^ \. 

^ - SSfiKv MfeCoKi, W Aard . F^rtonnM -Pe partM^tt V 

McGpiieuDDV* mrjoria, Ni^ York Ste|a \ 
pepartoent, o^^ivU.Serw ; ^ V 
K^S^iE, Ann©i New^ York Statt Depwt- / 

ibent of Civfl Servl^ J" 
McGumi, John P., New york' State Idu- ' - 

cation ^Deps^frtaient ■ ' 

MeKnmsv, Litoi Educ^tiQ^y ^Testing 
5s^c^^ ■ - 
MaL^uoKUDf; tilnrietibNT., FkridatStete 

Mql^^K W#e^% .J^teftaitba4 

ieis MaqhiniS C6^ration ^ 
Mbmtm, John V^. Uni^ersiiy of FloH^v, ■ 
MtSi^^Att* H., Brown IJjftivi^^ • : ' ^ 
MAGH&orf^Hfeq' W.,^Cdlfomi4..peparW'V, 

ment bf gducatibn **" ' 
Mal5^^, Janfi . ftiblic S^b^l System 

(Barndto,. Connecti^0# ^ ^ 
MANuiLj H.^T., ^University of Texas ' 
MaroA^^ K.;\^ur C^CiopFubiica-; 

tiona ' V* ■ . ^ ' ' ' ■ 

MawmottT, John Bostoh Uniidriii^ ' 
MiWfii, JMa^i Mp^aU^Ml T^ting^ServlGe 
Mabstqn, :H: ^M., ESl^tiiml ' Testirti 
^ • Ser\^i^^*5^ tW - Wt ■■ 

MAfl'WfrPrisclUa Clarkj |^ew Wrk Ci^ 



, Birtiardi 



Optical .MATipg^iOSf, Robert ^/Bf 



Educational Testing 



'V^Ser^ce - t. - .■ ^ -^.l. 

Livri dharfctte* Natidnd X^agiie for Nurs- 
' %g ■ ; " ■ — = ^ .■ ^. "•■^ ^ 
L^«jQiftay, 1' F;, Stat© UMye^ity of Iowa 
ltaTOi<i, "William 6.^^ Fund for tiie 

Advanc^'ment of Ed^tiitn 
Lm^^s, StahlAy, Teshple Ufciverjity 
^jOHMANj WaiiriOT/ A,i ' New York State 
■ ' Education Department' ^ "-™'r^„T' ^ 

Loj^a^ ]ylliim D-, Professiond Examination Metz, Elliot' 

liONO, Loi^ City CoUege of New^ork 
Lo^^ Frederic, Educational Testing Serv- 
ice 



EdSati^ ( Ne^ l^m fifiy) 



:d of Hipi er 



MAXVfe^LL,*Eleanor; 

land % a If; £5 '5^, ■ f 

MA*nfflw,''Lewii^.i Cooperative Study of^ji 
E^lfaaiion i^^ener^Education ; ^ 
MeJB^, ^,;^p^3dttcaSonal Test- 



irig^ervice - 
ME^BLsONi htotln A,, Test DeSilojiment , 

Division, Md^hflU Air ^r^ mme; N; Y. • 
Merry j Robert wr, ^Harvard Ui^er|l^v 
MESsiCK, S, J.; ^ducaUqp^ Tes^^l^ervic^ 



' JipTO Shirley, Educational Testing S^ice 
LoBGi, Ining, Teachers College, Columbia 
Univerdty . 
Ltmsi, Aidie, Amy Medical Service Grad- 
uate School 
LuoAi, ^^1^ M,, ^dar Crest CoUege 
hmm, Loulii NomaS^) Connecticut 



iueens^ College . ^ 
MiGHAELj V^bni Burton, University of 

Southem O^Cipua 1 jt^ 
Mil^^^phen^., Educational Tlsting 

MiLLETt, Esther, ^\^ive^'Sch^^ ^ 
MiTCHmLj Bl^tl^ C, Com- 
pany ' * ^ m^^i 
Mrram^ afeld,^Er^ qi^ ^legt of New 
^ York. . ' ■ - K 
Moi-LEKEOffFg Wi O*, Edn^^y&id Testing 
Servi^ . S 
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-.S^^^^i^ft^ - • -----••^-■^^^ - 

Nnji^ Katihs^ F^er,/^ihart and Gom- 

NoLANj Edwfird O g. Educationi Teitirig 
•-.Servle© -'^ -^ ' ■ • • ■ • 
NoLLi Victor Mlehlgah Stati CoUeg© 
.No^ooH^ Abrahani I^aw York State Da- 

North,, Robert D., University of Ken- 



NdsDw, Sig^imd, Michigan State CoUege 
P'CoHNOH, .Virgil hr Haadflpfiittters, United 

States Air Fore© V 4 
O'Kakb, Marianne, JEduqatiohal Testing 
, Servic© 

Olsen, Ma^orie, Educational Testing Seiv- 

ice :_ , • . . 

OM^ANSj Beatrice S., Biireau of Shipi, , 

Navy Deparbnent * 
Orleans, Tos^h B^, George Waihington 
School (New Yorlc) 



ORRi David, World Book Company 
Obshansky/ Bernice* Mitchel Air 
^Base 



OsnimoHEB, L^nardi Tlie GoUege of the 

Gity of New Yoric 
Pace, G. Robert, Syraeui© University 
Pace, Maureen^ The Piydiolo^oal Co^- 

ration 

pAijkm, Orrtlle, Educational Testing 
Seiviee 

pAjHiUJAN, Siroon, Queens College 
PiTOH; James B„ Vir^nJa State Depart- 
. ment of Eduction 

PEABsopf, Riehard, Educational Tasting 
Service* 

P^raAM, DoroAy T,, The Bancroft 
School * 

toLMAN, Mildred, New York City Civil 
Servic© Commission 

^TEMON, Donald A*> Life Insurance 
^ = Management As^cia^on 

p£i:EitsoHj ShaUer, AtDerican Dental Asso- 
ciation 

PTO T Tp g^ Laura M.; /Silver Burdett Ctom- 
l^y 

Priw, Hugh, J^rvtoi Unlvarsity 
FkE^Nj George A., Queens CoUege 
VmoA^ C. F,, Educa^nal Testing Sen^ice 



^uiou, L^ette k» Eduoationd T^- 

ing Sefdc©' ^ 
pjQt^i A, T«rrene©j Mitchd Air Fora 

.•Bsii©^:>^ /; . - 

PowMi<a, KbrtMn C*, New York State De- 

parbnant of OiyU Servic© 
]PotN, Editib Marjgare^ A© Psyclralogiad 
^i^ration ' 

ttlda, ^© Paycholo^^ ^rpdni- 
■ tion ■ . ;■ . " 

^Qvi^r^berty^^^niieri^n-Gouni^^b-Edu*- 
■ cation, . 

RAi^a^m^i William, Bank S&eet GoUeg© 
L ^f Bducation 

R^^^tBi, Jolm H„ OweM-lUinois Glass 

Q»mpany: \ 
HAi^d!0, 'Aomas J,, Hansard Budness 
' School ^ 
Regans James 8., Social Devices Center, 

ONR 

RffipmT, HoLroM C., Tempi© Univeraity 
Riccnm, Henry, EduMtiortel Testing 
Service 

RicHATOsON, Ruth P„ Richardson, Bellows, 



Rim, J. H. Jr., The Psydiologlcal Coipo- 
ration 

RiESSMAN, Frank, Tlie City Collage of 
New York 

RiMALo^H, Jack K., Edu^tional Testing 
Ser^e 

RiMOLDi, H. J* A,, Educational Testing 

Service 

RoBB, George, Educational Testing Senic© 

RoBBiNs, Irving, Queens College 

HocA, Pablo, Deparbnent of Education, 

San Juan, Puerto Wco 
RdESSLE, Robert L, B*, Tli© Standard Oil 

Company of New Jeney 
RoGERi, Miles S., HarvaiS University 
RpsEKBLATT, Frank, Cornell U^versity 
Roy, Howard L., Penonnej Research 

Branch . 

RuB^, Claraj Personnel Department, City 

of Philadelphia " 
RuLOH, iP. J-, Harvard University 
RuNDQUiST, E- A„ Personnel R^eareh 

Section, AGO 
Sapacca, Robert, Educational Testing 

Service ^ . ' 

Satt, Edward, Rensselaer Polyteclmic In- 

stitute ^ 
Sanfobd, Nevitt, Vassar College 
Saunders, David R*,. Educational Testing 

Service 

ScATOi, Ah^ Yeomans, American* fioundl 

on Education * 
So^s, Douglas E,, Univeiiity of porida 
ScHEiDEH, R, M., Educatiopal Testing 
Servic \ 

Schrader/ William B., Educational Tell- 
ing Sewlc© ^ 
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^ni^reNce ■ 

mint of Cii^ Serviae . ^mpmy 

.SHAii^, Catheripp Education^ Teitins 



Sl5Sr^%'* >larion F.j American Institute 

for Researgh .,,11** 
Smrman, A. W. Jr,r Mitchel Force 

SmNlLOOM; Charles, fhUadelphia Board 
of Pubfic Edueation . ^ 

ShimberO, Banjamta, Educational Tegtiog 
Service e 1 1 

SiMWSBUBY, Roy R.» Pingry School . 

Smtth; Alexander F,, University of New 
Hampshka > - • = . 

Smtth/ Allan B.> U^vemty of Connecticut 

Si^TH, Densel D., Office of Navd Re- 

SiJm^ Muriel, Educational Testinfi Ser^ce 
SoLOMOH, Herbert, Teachers CoUege, Co- 
lumbia Umveriity , „ 
Solomon, iRobert, Educational Testing 

So^™^ Mary Tayloe, Tower HiU School 
Spaney, Emma, Queens College 
SpAtn^ING, Geraldine, Educational Kec- 

ords Bureau , , ^ , ^ 
Spear, ArAur, World Book Company^ 
SraER, George S.> llUnoii Institute of Tech- 
nology _ , ^ i, ^_ 
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Ona 6^ j^fQM^ &i ftls fiianneri which is very closaly analogoui to 

^ W^ hsLVB fdund ttiat tte oonvergenca in several problems is fairly 
rapid. H ft© convergeni^ should turn out to be slow iera tiiere is np 
iflrigft fa rwfl^ng Mit flf^ It^mtlnns. If next 

two dmrasiOQi are nearly circidAr^ so tiiat the next two latent rooj| are 
nearly ^© sama, than tibe convargenoa ii Imown to ba slow, \^en ttiat 
faappmsy wd ihould be pmctical enough to reidiia that it really doei not * 
m^a -aifty differenee where you put the n^t axis. You cwi stop after 
about tiha Qifril It^tion. If you are taU^g to a mathematicifn, he 
might not aoeept It^ but for pmetioal worK it is useful. 
^ ■ to raise several spiecifle questioris* Oner if 

we are gobig to do tiiis on the least square basis, what is the philefsophy 
tiiat guides us to usa unweighted ^uares of deviation P If we were 
looking solely at iampl^g varlationps which, we are no^ then we woi^d' 
want to weight titds quite a lot diffefenfly for large carelatipni than 
? small. For the pu^^se of factor analysis as a wholes I am not clear ho^ V 
^#e shoiUd wai^t it. I am just asto , 

Dni Thubstdne: You are not referring to the A's as weighfa. ^ 
Dr. Toteys NOj I am referring to the possible weights outside tiie 

parenAaste^, vliAM'^;V:v . * 
..^ Ita^irt^roifl^ to raise is AisV if we are going to 

give up ^e communlahties and go ahead in this direction, putttagtaside 
ie dfflculties of mathematics in computation which might^Jbe really 
seriouSj if we decide to take out two factors, why should we lake out for 
the first one, die on%ttiat comes first and singly by this mettiod? Would 
we not be better btf, in principle, maybe not in practice, to accept this 
fonnuld and xnin^dze with respect to the A's and the B's at the same 
jdme? If OT^ 0ould do this, in this Example you cited bf Rank R, you 
wotjjii ggt:^i\^erytWng^tf could do it for all fi's'tbgeUier, _ 

Dn; TM^aiTDl^rl^s,!! wpU rather write the equation with R plus 
one or two or diree taraisrYou could then discard some if you want to. 
However, suppose you gamble that tiiere are, say, ten factors. You 
write tan terms. It gets a little unwieldy to handle* TTiat js why we do 
not do it. ^ • T , 

Dr* TuoictI I have one question about the examples ^ou have tried* 
Did some examples have somewhat equal commurialities while other 
samples had quite variant communalities? I suspect the range of 
fcdiSunimahties makes quite a bit^ef difference in the closeness of ap- 
pro^dmation by your metiiod. If die, faommunalities were to come out 
die same, your method wo^ild yield an exact solution. : * 

Dr. TteraSTONEr That is a good point to make. As a matter of factj 
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we s€t up sev&tBi mplrieal cai^, For im^ variation m communalitiesi 
cpnvpi^mc^ was mu^ mcffe rapii In a case witii wldt difference 
of ^mmunaUti^ ft a conveFgence was slower* I think you are quite 
right in Aat resect. ' ' 

1 .ij^p.Ijq^ HjPoi^pQtot«Uo ^uival en t to ^ iitfng 

zeroes in fte diagon^* What ib fte advantage in having zergei in fte 
diagonals rafttr ftan onesp Is it an approximation to iometfaing else? 

Db. TmmmNmi That is not tiie same problem. You are raising &a 
question of why we should use commurialify instead of unity in ^e 
^agonals, / - , 

'Dm* homi No, If^r^i are not going to use the communalitfieSj,.^^ . 
you better off uitog l^es rather than ones? ^ ' 

D», TmjEiTONE* My objective is to ign^re^^die diagonals, and diey do 
not participate in the equation, Tliat is file principle of ^|^metiioa, 
If diagonals aremnknown, let us leaye them blank We ^ not have 
tb^wite zeroes— Just leave themj blank, they do not participate. How- 
av4r die computations would be the same as if the diagonal correlations 
were zero. - 



4'- ^ 













SEGTION 2 






Eyaluating Group Interaction 












"."[r^- " — -— — — — ^ . . /. ' i ; /'••:y . - 





ERIC 



67 



^7 



-I 



TESTING PRpBLEMS 



4-; 



' A New Technique for Measuring 
indiyidual bifterenie^ m Confo rmity 



to 



gmeht 



RICH AJL D S . CRUTCH F IE 




Centeal to resear^ bofli m.peftbnalily aJia"ta;gr§i#^^^^^Pflf are, 
meth^ for tte mdasuremeht of tiie indivia^aTr to^iprifi'si^ 
of group-interaction, for exainpl^ conformity -beHa^^Jft' to g^^^^res- 

sure. • ;, ^ '■ \ ■ ■ ; ■ ■ 

Su^ measurement is made difiacult by certain demanding require- 
ments. (1) Idealiy, tiie behavi^ should be measared directly in actual 
group situations. f#flier tiian iBdifectly by questionnaires abput group 
iituations. (2) The.^gfoup situations should be psyohologlcally feleuant^ 
for Ae individual. (3) iTiere should be standardimtion of the ^oyp 
situatiph so tiial'measureniflftts of diferent Indiv^duars may properly 
Jae cbmjMired. And (4) Aese should be adequate economy of the.test 
me^od, so Aat substantial humbers of persons may be tested without 
unreasonable cost in time and money, ' , 

' the stindardizatlai problem Is the most acute., In genuine groups. 
Involving face.to-facq(kint#actiori of several subjects, the stimuTus 
sltuatiori confronting eactf person is unique, being dependent in part 
u|Son what aie others In die group do. This leads to an undesirable con- 
founding of peraonal and situational factors In the measurement ofthe 
.individual's CTou^behavfor, and there is no simple way to disentangle 
'the two. 7 . ■ * 
,: .One fruitful attack on this problem is what in previous research 
'applications I have called the "quasi group-iijterpction method." Its 
essen^B is simple. Subjects to be. measured are placed together in a 
group situation which, though perceived as genuine by them, is 
actually so contrived fey the experimenter that he wholly controls and 
manipulates tiie conditions of group-interaction. This serves to stand- 
■ ardize Ae situation ifor each subject In an -identioal fashion, so that 
flbserved individual differences in' behavior may properly be ascribed 
, %'4lffCT«iCes wnong the persons rather than to,sltuatiBnal differences. 
" The quasi group mediodolo©f is by no means new. Earlier variants 
are found throu^ut the field of experimental social psychology. 
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LiMe, however^ has been inade of it in personality measurement, and 
j in general ffia meUiod has not been widely ej^loited. 

In recent work I have ^plied it to what gives promise of being a \. 
powerful new tedmique for measuring individual differences in con* 
' {pm ^ to group Jud pne^i F ive parsoni are te sted iiniultan0Quily> 
Hey ar© seated in a row in front of eta elecfrical switchboard appa- 
ratusp consisting of fivevidenttcal panels. Eacfr panel is shielded from , 
die o^er four* It TOntalns eleven numbered switches by which the 
person baay signal his reiponse^ .It also ^ntaias five rows of signal 
li^ti whi^ can display the re^tases being ^ade by the other four 
group maaabei^. In shor^ there is a simple electrical communication 
syst0m among the five ^rsons. No direct coinmunication is peiroltted* 
The task for each person is to make Judgments perteining to a large 
ntiiiber of slides which are projected on a wall facing the group. Each 
slide offers a set of riumblred alternative answers among which he is 
to choose. He records his choice by closing the appropriately numbered 
switch and tins causes his response to be displayed on the panels of 
thi other four members* He is also instructed to respond in a specified 
serial order within the groups person A going Rtst, tiien and so on, 
^ E being last, ^e designation of his letter position^ — A, C, D or E— 
is indicated to hun >hy the experimenter. Such letter designations are 
rotated from time to time, permitting each person to respond in each 
of the five serial positions. The experimenter offers no further explana- 
tion of the/purpose of this procedure. , 

TTie sUdes offer a mixture of materials— simple perceptual compari» 
sonSj sudi as of relative lengtii of lines^ Zog joa{ problems, such as the 
completion of a number series of the kind found in standard mental ' 
tests; expression of one's own opiniori^ and attifudes-on various issues, 
etc. % \ 

On the first set of slides the person finds the judgments fairly easy k - 
to make, and he observes that there is a sensible agreement betw#n f 
his judgments and those of the other four members. But when later in 
the series he is for the first time designated as E, so that he must re- 
^^knd in last position, he experiences something new and disturbingi 
this slide, calling for a simple judgment of relative length of lines, 
he sees the other four members unanimously agree on a choice which 
clearly confradicts 1ms own* This throws him into a severe conflict be- 
tween the clear evidence of his own senses and the unanimous contra- 
dictory consensus of the rest of the group. How he chooses to resolve ^ 
this c&nflict, either by yielding to the group pressure and confonning 
to its judgment or by r.emaining independent of it, is the basic measure 
of conformity behavior in our procedure. 
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^ of ^to one audi critical slide, VV^e rasped- 

^j^^-p^^tiimMf h© is p^en^ wth more Aan ,bventy otiie^ CTiti^r 
sUd^^p^diiAiing ' y^oiily tb mattaj^ perception^ of logic, of 
o^liiion^ and of tfttituda— on ea^ of which ha iU conf ronte J witii a 

of Uie ^up. \ . : ^ ■ . ' ■ V , 

/ iy hava sunnis^d, the siturtCohJ^nd^ ^eaUy what 

^tbi ^sotai hi^ to.^derstarid; TOey hpe Jbian grossly 

; deceived^ ^e five ^^p^ are not connaetad to onfrano^er af aUj but 

to a oontrbl l^ard operated ^]^^ Mpiiipfltanter, It li he who signals 
'the r^pQflflB^ j^l^j^ frpjn die p^er ^oup jneniberSi 

The yni^ Adt die. pattern of signa|i he choosarto 

J muliito .'appears: simultanaouily Mid identically on/ all five panels* 
kloreoyer, tte deslgdationi of Trrirtllnrrirr of respoijding— A ttirou^ 
E--^ve likewise Identi;^ jFor all five plraons at .every moment 

By /this c^^^tioti, tii©rrfpre, a quasi gpoup-interactio^ situation ;has 
; bean confriyedj pe^ittin^ eadi person to be expdsed^to a standardized 
set of simulated group judgments and to be confronted at jredeter' 
£nined po£^ts, witK Identical external pressures toward conformity, 
r Since the Spring of 1953 when dils technique was flrst developed,' 
three Jitudfes have beemnaSe using different populations of subjeetsV 
i Th© first study ^w^ tfveraging 34 years of age, all members 

of a prof es Won in which leadership is one of the iriogt important qualifl- 
daBons/ 31ey were tested at InsUtute of .Personality Assessment 
jmd llaseareh as part of a larger assessment program. The secpnd study 
#as of S9 coUege'stiidentSp- mbstly sophoinorei. The tiiird swas of 50 
women j all college alumnae in their early forties being studied under 
Ae auspices bf die Mary Conover Mellon Foundation, 
, The results of these studjes» which attest to the technical success of 
the method and tiirow light on the nature and determinants of con- 
formity, behaVioif, are summ wito/great hrevi^ in the following 
ei^t'poiata; 

yl % Th$ gmwla conformity behavior mhibited fa farge. 

.A, single' fairly, representative^ item will serve to illustrate.. A circle and 
a star are exposed side by side, th# circle being about one-third larger 
' in ar^ ,Aan the star, Tlie false group consensus is on star aa the larger, 
apd a WW^^ ®f A® 1^6^ 8*^® t'^^^. sarne false judgment* 

Tft^ dftgf^^ pf ^fi^fmt^y shown depends in part on the^ind of 
nrnterial biing fudged. Alttiough some amount of /^onforntity can b6 
alidtad ■ bfr evmy one of the 'Critical items, tlie range in OTectivMess ^ 
among them is Extremely wide, ;At; the lower end, a .simple expression o£, 
personal pref ermce ioT oi^ of tti^p line-drawings is very^Uttle su^cept^^ 
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ble to a contradtatory group consensus, dagree. °f c^n^ 
mg from zero to ten peroent in the several studies. At the upper end 
^ nit effective item is one whl,h has been dehberately maximized 
for ambiguity, Tlic subjects are^sked to complete a number serws^ 
or ^ich^lre is actually no logical sohitioi. When 
group judgment agrees unanimously , on an obviously teal com- 
nletion 79% of the men express agreemenf with tins anfver. . ■ 

3. Substantial conformUy is mcited onseaiam, importrnt lud^nts 
as wettas Qn more abstract iudgmerits. A critical methodo olical issue 
is whether the conformity effects so far mentioned merely represent 
rather superfloial readinesses of the person to agree with, the group on 
matters of little real' importance to him, or whether instead thej do 
reflect more basicgsonformity. tendencies in the person. Swpport tor the 
latter interpretati4 is found by the introduction of newxri^ical items 
in the studies oflcollcge students and , mature women. These items 
called for expression of the persons attitudes on matters of civil Jiber- 
Ues subversion, ethics, crime and punishment, etc. Pronounced con- 
formity effects are found on these socially and psychologically relev^jnt 
items. Take a single example from the study o^college students. The 
question was asked, "Which one of the following do you feel is the 
most-important problem facing our courttry today? , and these five 
nltematives were offered: economic recession, educational facilities, 
subversive activities, mental health, crime and corruption. Among 
control subiects tested alone," only 12%' chose "subversive activities 
as the most important. But when exposed to a spurious group consensus 
unanimously maUrig this choice, 48% of the subjects gave this answer. 

4 There are pronounced individual differences m amount of con= 
fmmUy shown. A total conformity score for each person may be read- 
ily computed by summing the number of the critical items on wluch 
he exhibits conformity to the false group consensus. Virtually the entue 
■ possible range of such scores is found in all three studies^ Among the 
men foypstance. at the'lowcr end several subjects showed practically 
no ^&ity. being influenced on one or two tens at the most_At 
the upper end one man was influenced on 17 of Si critical items, The 
scores are well distributed between these extremes, widi a mean score 
of about 8 items and a tendency for greater concentration of scores 
■ toward the lower conformity end. As estimated from tjie correlation of 
sub=scbres on two matched halves of die critical items, the reliability 
of the total score for the sample of men is .90, , 
' 5 There are both generality arid specificity in the conformity tend- 
encies among ln4ividuals. Although there is a genernUy positive matrix 
of intercorrelations of degree of conformity effect for the various items. 
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ftere are also some useful differentiations t9 ^e made among^e 
A cluster analysis yields ^pne principal set pertaimng t^l^Uy stud- 
tured items, ie., those* 'involving fclear judgments and unambiguous 
stimuli* X second Aain clustef^ consists of poorly strudtured items, 
iie*, Aose invQlying uncertain Judgments and ambiguous stimulil ^e 
individuals can be scqred separately on Aese clusters* There is evi- 
dene© ^at tiiese cluster scores ^as, well as individual performance on 
single items must be taken toto account In ^e analysis of «conf ormity 
behavior* y ' . ' , 

6/ THe degree of coi^Offnity behaviar felates significMly^o relei^ant 
d^eniions of persomlUy, Validity of the conformity meisure %£^^^ 
tested to by qts substantial relationships wi^ numerous ratings^ osf^c- 
tive test Scores, *and other personality d^erminatlons in agessment of 
tiie sample of men. Thdse low in con^^nity, 4at iSj jfliose Wbo sitfcess- 
fully resisyriie^oup pressu^, ^n be clearly characterized ^having 
intellectuareffectivenesSj ego-sftrengflij self-acceptance^ leaderehip abil- 
-ity and maturity of social relations. The high confonripis, on the con- 
traryj reveal iijferiority feelings, rigid and excessive self-co^b'ol, intol- 
erance of own impulses and lack of self-insight^ authoritarian outlookj 
emphasis on external ^hd socially approved values, arid disturbed 
attitudes toward o^er people. This genepl picture coincides ^ell ith 
j^rior theoretical and empirical studies pf^e, personal traits ^associated 
with cortformity, butj of course, it repre*^ents a gross oversimplification 
of die complex relationships yet to be explorf^. 

7, There are significant differences in merage amount of confomiit^ 
exhibited by the several populations of subjects. The coH^e student 
samplerwas made up of both m^les and females* The male students 
exhibited just about the same average larel of conform^ as previously 
found in the adult, professional men* But the female students exhibited 
significantly more iconfonnity than these male groups* -Ofi^ the other 
hand, the sample of mature women— cojlfegf^ alamnae in their early 
forties—showed signiflcantiy less confbrniity on the average tfian that 
foiind in all the otiier groups* ^ 

8. Expefimental variations in the testing situation can^hajtge the 
amount of conformity shown. We have seen that tfiere are differences 
in degree of conformity relating to personality detenninants, to the 
nature of the pgpfifations of subjects, yjd to the kind of iteWs being 
judged* We can also show differences resulting from certain experi- 
mental changes in the situation itself. As one extopje, when a group 
of student subjects were exposed to additional pressure, namely, that 
given by having tiie experimenter later confirm the "correctaess*' of the 
false group consensus, conformity pff^ts were markedly ipcreased. 
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To reyert now to our ^itial 'Statement o£ requirements for proper^ 
fl^easures of individual "behayip- in situations of group-iriteraction, 

^ namely, directaess, relevancf , standardization and economy, it ^puld 
appear ^at present method fulfills all these requirements. The 
very essence^f &e method is such as to guarantee standardization of , 
conditions. It is c^fely economical, providing for tiie testing of five 

^.persms It once in period of approxiniately one hour, It is direct, in 
that^e persons perform*in a situat^n which is for them pne of real 
gr^-interaction. dn this poii^it should be emphasized that the genu^ 
inenesi^;i?f the situati|« is v^irtually never challenged by the subjects. 
Of tiie^fel of 159 person^ alAady tested in the procedure, only a small 
:hanaful wh^toiestioned immediately afterwards expressed doubts. 

\|6f Its genuini8e|s; of these only two or three seemed to have |elt such 
doubts wl^e ^actually in the situation. ^ . 

Finally, as to Ipychological relevance, we may Judge from both the 
mantfest b^avior and the retrospective reports by the subjecfe that 
th& situation was deeply involving and that the group pressure crelted 
an^riety, often ^ute. A substantial number of persons freely admit gn 

♦ later questioii^ng that they had violated ^their own inner convictions 
jn ord^r to give responses in accord with those of the group. 

The findings 'ktffest to a ridi potential use of this measurement tech- 
nique when applied to a variety of problems— personality assessment, 

^ ^Qup dynamics research/' sociological fend crosS-culturaLcoftiparisons 
of g^gupsi experimental and the^etical study i^f the psych6logic|L] con- 
ditions of conformity. A dumber of such studies are contemplated and 
several are QQW in progress, ^ 
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Thf Russeft Sage Social Relations Test: 
A Measure of Group Problem-Solving 
' Skills in Elementary School Children 

DORA E. DAMRIN 



TTia test I am gomg to describe tb" you this morning represents an 
attempt that has bean made by myself and certain of my colleagues in 
ETS to develop an instrument of measurement that will get St certain 
of the intangibles In education. As all of you knoWj over the past sev- 
eral years flier© has been incrtasing emphasis^ particularly in the field 
of elementary education, given to the idea that chil5ren should be 
taught to work together cooperatively^ to share things with each other, 
to participatg in the planning and carrying out of certain classrdom 
activnities, and to conduct themselves in ways compatible with a free 
democratic society. ' ' . 

The test I am gofrtg to show you today ik designed to measure the 
extent to which some of these things have been^'accomplished, Since 
the test is a little unusual, I thought that it would be best if I would 
nretend diat you arfi' a group of students and give It to you in the 
sa&ie way that we give it to elementary school classes. The score sheets 
yomhave will be explained after this demonstration, 

Tme examiner enters the classroom and ^ts up a card table in the 
froni of the room. He then says, "Today, boys and girls, I am going to 
sive'you a test. Now, you knw usually, when you take a test, each, 
qne of you sits in his own seat, does his own v^ork, and you are not 
allowed to help each other. Isn't that correct?" ■ 

The students usually nod rather solemnly in the affirmative at^is 
point. The exiimiher goes on to, say, "But this test I am going to give 
you today is exactly backwards. In this test you are supposed to help 
each other, you are supposed^ to work together, and you are supposml 
to fflk to each other while you are taking it," Here the children usju- 
ally evidence some amazement and seem to wonder what in the world 
Is comTing next, . , ' " 

The fxaminer then says, "Let me show you the first problein on the 
test so that you will understand what it is about." The examiner opens 
the testing case and holds up this rather at6'active form, built of dif- 
ferent colored plastic blocks, and say'S, "Now, what does this look like?" 
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The children resplbnd withj "Thats a house/' f^, 

The examiner continues, "Yes, that% right, this is a house, and I am 
going to put it here on the card table for you to look tat all duV|ng the 
time you are taking the test." 

"Now here in this box, boys and girls, I have all of the blocks that 
you will need to build this house, and what I am now going to do is 
pass around the room and give each of you one or two of these blocks. 
Then I want all of you to get together and see how quickly and how 
well you can build this house " ^ 

The examiner goes around the room, giving each child one or two 
blocks,, depending on the number in the class, and at that instant we 
start scoririg the test, continuing on until the children have completed 
the problem. 

After the blocks are passed out, th^ children are told, "Now before 
you start, you can take all of the time you need ^o plan how you are 
going to get together and do tliis, bub after you tell me that you are 
ready, then I am going to begin timing you, and you will be given 
fifteen minutes in which to build the house. If you coinplete your 
house before tliat time, your score on tlie test will be much higher." 

The examiner then works with the children while they build a plan, 
and all during this time their comtnents and suggestions are scored. 
When the children indicate that they are ready tcr begin, the examiner 
scores them on how well they carry out their plan. At the end of the 
first test problem the examiner presents them, with the second, and 
finally Witii the third problem. 

The second and third problems are progressively more difficult 
than this one. If you would be interested in examining them, 1 will be 
glad to show them to you afterward, since our time is somewlmt 
limited. 

Unfortunately, in my discussion of the scoring of the test, I cannot 
present data on the validity or the reliability of the instrument. It is 
^ of too recent developm^^nt to do that. We are, however, presently con- 
ducting a major validity study in cooperation with Dr. Orleans and his 
research staff in New York City. 

On the basis of the data which will be collected within tlie next 
'six months or so, I hope that 1 will be aljle to support some of the 
things thaj^ I am going to hypothesize about this morning. 

The test has two separate and relatively independent parts. The 
first is called the Planning Stage and the second, the Operations Stage. 
The Planning Stage begins at the time the examiner starts passing out 
blocks to the children, and ends when the children infonn the examiner 
that they are ready to begin constructing the model The Operations 
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Stage b^ins when Uie examiner teUs tibe children "Begin" and starts 
his stop watch, and ends when the 15 nunute time period has elapsed 
or when the ^fldren flnish building tiie modeL A set of six scores is 
obtained for tiie Planning Stage and a set of diree scores for tiie Opera- 
tions Stage. 

The first score given In the Planning Stage is temied Group ParticU 
pation. It is based upon the four speciflc, behaviors listed on your Ob- 
servation Sheet at die <opf of the page. TTie Observation Sheet is 
checked as follows I * 

li Individuals A check is entered each time an individual class mem- 
V ber makes a problem-relevant compient* 

2. Buzz Groups: A check is entered each time one-half or more of 

the class members spontaneously turn to each other in small 
groups and begin discussing various elements and aspects of 
the problemi Many children speak at one timej comments are 
not Individually distinguishablei Wat it is cWar tiiat tiie chil- 
dren are engaging in problem-relevant remarks. 

3. Chorus: A check is entered each time one-half or more of tiie 

class members spontaneously chorus^ Iheir opinion about a 
previously made individual comment. This chorus may indicate 
eitlier agreement or disagreement witli tlie i^a. 

4. Noise: A check' is entered each time one-half or more fcS the clas% 

members become noisy and undisciplined. Mafiy chiraren speak 
at one'timej the examiner is unable to hear individmal comments, 
but it is clear that the children are engaging In many problem- 
irrelevant remarks* i ^ 

The pattern of checks made by a group of children over tiiese four 
categories of behavior is translated into a numerical score which re- 
flects the amount and quality of the childrens total participation* A 
high score ton tins variable indicates that all members of the group 
participated actively and freely in the planning stage, whereas a low 
score*^ndicates a minimum of free participation. It is our hypothesis 
that the Sroup Participation Score is closely related to the amount of 
freedom which exist! in the classroom of a particular elementary school 
teacher, 

The second scor^is ^Communication Pattern. It is concerned solely 
with^e intent of the verbal commf nts made by the individual children 
who contribute to the planning dismission and who are checked in the 
'Group Participation Score. All comments made by individual children - 
are classifled in one of the four^atagories>ltsted second. Ai your Obser- 
vation Sheet. These categories are deflAed as follows i t 
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1, Suggests discrete id4i A ^ck is made if the idea is one which 
\ . has not pre^dously bieen stated and which does not Lncorporate 

any part of or build on a previously mentioned idea. 

2, Repeats ideaf A check is made if Ae comment is either a rephras- \ 

ing or a restatement of a previously mentioned idea, 
a Evaluates idea: A chedk il mada;if' the comment t 

mentioned idea ibmt^ out why It is w^ong^yhy^tt wn't . 
' work, or why it is a good sugglstion and should ^be -adoptej^. 

^ 4 Improves idea: A -check is made if the comment takes a previously^ 
mentioried, idea and builds on to it so that the end produgf is' 
more cdmprdimsive, more precise, arid,mor^ detailed diafl the v 
initial idea. ^ , \ * f 

The pattern of checks made by a groiip over these^four kind^ of 
statements is translated into a numerical score which r^geets the way 
in which children converse with each other in buildlrig ;aV^tyally 
agreeable plan of action,. A high score is obtaifaed.by Wife group whose 
members improve and build i^on each bther^s id^QS and progr^s^ 
work toward a fi^al plan that Is more precise^ more detailed, anB niore 
comprehensive ffian the initial idea for a plan wiJdi which the^group / 
started A low score is obtained by the group whose mOTibers%cQm^^^ 
ments are largely discrete and repetitim In this score a: s^rie^ of dis^ 
Crete and repetitive checffi' indicates that chfldren are comj^tlng.with 
each otiier to give "different'* ideas rather than coopefating^i& each , ^ 
other to build a consmictiva, solution. Tlius the signiflcanc^ of the 
Commumlcation Pattern Spord resides in the fact that it provides ih- 
formation ibout whether the members of a particiilAr ^erneptary 
school classroom function as ^ collectipj]t.Qf dngl6 ihdi^idOals or^^ 
cooperative group. , ^ : 

The third score is tmme& Orffmizatlonal E^crenGi/ and^is;^ 
upon the content of the ideas which -children propose' during the 
ning period. Fortunately the n^ajb^ ideas which children ^ 
about ways of solving the^ test pppbleifns are finite^ and all p<^^ 
gestions can be classified into .dnp ;of ^e ten categories listc 
third section of your, Observatidn Sheet. These categories 
following deflnitioris: ' g \" . 

L Construction Authority; Having orie or a few children 
all of tihe actual building . - [ 

2. Supervisory Authdri^r Having id^e or a f^. children s| 

and help othersA^hd are building the^u?!zfe \ ' ^. 

' 3. Simultaneous Groups r Orgahizing the^class toto groups ^on 
' ■ basis i^Ad having all groups working simultaa^sly ^ . ^ 
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4 S^uential Grmips: Organizing elais into 



Md spa€if)^ng tibe order in wfilch tiie groups are to take tui^-J^yi 
in buildirtg the puzzle -^^^^^^ 

5. Ttoblem Groups r Organizing tiie class into. jroups on tha;A»fe^^ 

of some itmcturd . element of the puzzle^ e.g.^^ ^ 
a; By: eqlors: red, white, blue 
b. By parts r neck, body, legs, tail 
c* By shapd of block r square, triangle 
d. Any cbmbtpation of the above .j,. 

6. Non-problem Groups: Organizing the class into group^f^^thB: 

basis of ipmediing eWf ma? jo fte puzz ^ 
a. By sex* bo)^, girls 
b* By seating airangement: rowSj tables 
c* By ,numerica^groups: l-at-a-time; 2-at-a-time; 3^^l^^&ne . 

7. All at onoei Having everyone in the room go up to tl^' ^fc^l^ 

table at one time, each person putting in his own pi 
: 8. Same as last time: Havdng tiie group follow Ae pl^J^^^_ was 



^' f oUowed in the preceding probliem 
§/ Group disoipliner Making suggestions about how tb^jgri|, 
i fo behave while building the puzzle, e.g- T ^ 

a, Don*t push ^ ' ^ 

^'- -L .: V b* Don't crowd around V ^ 
, ^/ el Oqn*t jerk pieces away fr^^'to^^er peop^^ 
; " Change persOTneh Suggesti^ builders, n^#Sd^^ 

gesting .a d^erent order of s^uential groupi 
.The drgani^ti^d^ EflBciencyi Score provides information abou|,lic 
^much .knOYi^liifdge group has regarding ways and means^f org^' ^ 




^ fd attft!^ a ;^ftrticular problem. It will be observed 




^b^dtims 

' '^S'- 

Ae next 
and when 
^e class can 
nee" represents 



i0h tl^ list;^ Qjf^luzational Techniques are^ on fli^ wJ 
V m'jfleSeeii^ of^efficiency. The first two items 

^^^'d^ositlng leadership authority in certain groti^^ 
v^t^o^deal witK the idea of how the class shou^ 
* if% brgaftlzed; the nfext two deal witfi the baseS 
, uleAo organize |^lf inta^^i^upj; and the Idea "all ^ 
' coinp1pt# la^ of organizTOoh= The last three .items oK^e list of Organ 
^ izatiowl T^chiil4ues ,apply ^ly in the seeonJ and third problems. 
"V /Shese ideas, if checkedrtjnust be ev|f|uat6d in tenns of die plan used in 
^le prec^ing'problem.tf ^ ' & . 

^^The C^ganiz^onal EflBciency Score, if hi^^'|^^cates that a clas$ 
i . r6Qpgniz|s the nSed for leadership and for sottM^dokkU of ananging its 

" V' ttiefcabei^ mto a^ stincture diat will>f unction efficiently in the present 

y-M^m-t' :.^^-.J . .... .^..j....,,..,, ...... 



'^w^^^fei^lwi sJttffttiQn* A low score indicafes that childyf^lack neces^ 
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sary Imowledge of prganizational procedures and can ^nk of noAing 
better ttian to have each diild put in his own piece himself. 

The fourth score is temed Task-Centeredness and is based upon the 
same list of ideas as is the Organizational Eflaciency Score. However, 
whereas Organmtional Efflciency concerns all of tiae suggestions 
diildren have about ways of solving the problem, the Task-Centered- 
ness Score is based upon only tiiose ideas which are selected and in- 
co^orated into die final plan. The logic of diis score resides in the 
notion that a group skilled in techniques of cooperative group action 
will select, from among all tfie ideas presented, those few ideas which 
are '^best** in terms of the demands of the particular problem at hand. 
A group which is not skilled in Aese techniques will tend to disregard 
"good" ideas in favor of poorer ones. 

A high Task-Centeredness Score indicates that of all the ideas a 
particular group of children had about ways of solving the problem 
the '*best'' ideas (e.g., tiiose highest on the list of organizational tech- 
niques ) were selected and incorporated into the final plan. A low score 
indicates that although a 'group had several good ideas they rejected 
tfiem in favor of ideas which capitalized upon individual desires and 
mLnimized the total group good. 

The fifth score is termed Inifependence and concerns the group's 
ability to plan by themselves without external assistance from or inter- 
ference by the examiner. The elements of this score are basad upon 
the kinds of actions the examiner is forced to take during the planning 
period and these are listed in die fourth section of your Observation 
Sheet. 

The first action, "Prods Group," means that the individuals^ in a class 
respond with stony silence to the examiner's initial request: "Now what 
ideas do you have about ways the class might go about solving this 
problemr When this occurs, tl^-examiner repeats certain of the rules 
and the demands of the test anCagaIn asks, "Now, how are you going 
to get together and build this model? Does anyone have an idea of 
how the class might go about it?" "Prods Group" is also checked wheri^ 
ever the examiner has to urge the group to give more ideas after the 
first one or two. 

The second action, "Suggests Vote/' means that the class, after pre- 
senting a few ideas, comes to &n impasse and does not know how to 
proceed. That is, children keep repeating ideas that have already been 
presented, or they again fall into a complete silence and look with be- 
wilderment upon the examiner when he asks, '^Which of these ideas 
are you going to select?" In either case the examiner suggests that 
they vote on one of the ideas which they have presented. 
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• The tiurd action, ^'Reminds Group," means that the exammer is. 
forced to remll to 4e class the rules of ''good group behavior" Tliis 
is nec^aty when the children become noisy and undisciplined, when 
Aey ail taUc at once or shout out tiieir ideas to tiie examiner. In sudi 
a case the examiner is forced to say, '*Now, boys and girls, lets speak 
one at a time for Vm afraid I can t hear your ideas and I'm sure you 
€*an't hear each others/' 

The fourft action, '^Summarizes Ideas/' is tajcen by the examiner 
whenever tihie class reaches an impasse because of the complexity of 
their ideas. That is, the discussion eventually comes to revolve around 
^ee or four different plans, mdh of which is so involved and detailed 
' that the class cannot pull togetiier and hold in mind all of the different 
elements and cannot come to any "kind of decision. In such an instence 
the examiner names the plans in a one- two-three fashion, briefly sum- 
marizing Ae details of each* ' 

The fif A action, "Conducts vote or election" means tfiat the class 
has suggested that tiiey vote on a plan or tiiat they elect a leader. The 
^aminer then proceeds to perfornjthe mechanical tasl; of conducting 
the vote or election. J l ^ 

A high Independence' Score indicates that a gioup has the ability to 
discipline themselves and to carry on successfully a discussion from the 
beginning to a point at which decisive action is taKen. Gonvers^y, a 
low score indicates that a group lacks the several skills necess^y for 
independent planning and must rely heavily on oytside help to reach ^ 
a point where decif ive action can be taken. 

The sixth and final score which is given during the planning stage 
is tenned Decision Method and refers to the way in which the group 
decides upon a particular plan of action. These four methods are 
shown in the fifth section of your Observation Sheet. 

The method of active consensus is that in which all members of the 
^^group come to a more or less spontaneous and mutual agreement about 
a plan of action. The method of active vpte indicates that the class is 
split upon two or more, suggested plans and must use the method of 
voting to achieve the kind of consensus necessary to grouii action. 
The mediod of passive vote indicates that there is no real concern on 
the part of group memhers about which of several plans of action is 
adopted. Voting is artificial and meaningless and tends to be engaged 
in as ^somediing we are supposed to do." The metiiod of passive con^ 
sensus indicates that the plass as a whole is uninterested in botii the 
problem at hand and the plan which ougMt to be selected in resolving it. 

Our hypotiiesis is that use of the method of active consensus indi- 
cates a warm, cooperative, cohesive and skilled group, whereas use of 
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the metliod of active vote indicates a skilled but less cohesive one. 
Use of eiUier of tlrn passive methods seems to be associated with very 
low degrees of skill in group planning and group problem-solving. 

The scorm obtained during the Operations Stage have not yet been 
as preciselPdefined as those given during Uie "Planning Stage. The 
reasori is that the behavior which occurs during this second. stage is ex- 
tremely rAore coinplex and far less "controlled" than the behavior in 
the planning period. 

The first score in the Operations Stage is termed Execution Pattern 
and indicates whether and how well the group puts its adopted 
plan of action Into effect, once the examiner has said, "Begin. A higii 
score on this variable indicates that the children move easily and effi- 
■ ciently into the mechanics of their plan and proceed Without hesita-^ 
tion to caky it out in detail. A low score indicates that the plan breaks 
down immediately and that children rush up to tbo problem table 
in'a helte^skdter fashion with absolutely nd-Jtegard for the rules ot 
aroup behavibr they decided upon during their, glannmg session. 
" Our hypothesis is that children who are used to f*jinpting the rules 
which are to govern their behavior and, more important, who arfe ac- 
cistomed to acting in accordance with these rules will score high on 
this variable Conversely, children acclistomed to engaging m group 
planning as a relatively meaningless mental exercise will tend to^ex- . 
hibit some skill ifi the planning period but will fail completely m their 
ability to actually^ govern tlwir behavior in accord with the rules they 
themselves have outlined. , . , , , 

The second score on the Operations Stage is as yet a higldy sub- 
jective, andiaccordingly, n highly unreliable one. It is a measure of the 
overall psychological tone-quality of the group engaged m working on 
the puzzle. The descriptive terms li.sted on your Observation Sheets 
ire theckcd on the average of once a minute by tlie observer on the ba- 
sis of his judgment about what the group is like. It is our hypothesis 
tliat the "psychological tone" of the group is to some extent related to 
and hence indicative of the psychological quality of the classroom 
-itmosphere which is created and maintained by the particular teacher 
involved. Our findings here are suggestive, but as yet so tentative that 
it is not possible to discuss them in detail at this time. 

The fimd score obtained during the Operations Stage is termed 
Grnup Interest and piovidcH an Indication of the concern for the prob= 
lem Which is e.xhibited by tliose children who are not actively engaged 
in the construction ta.sk. We are not completely sure what is respon- 
sible for observed differences in interest in this test between various 
classroom groups of children, but we think that the group that mam- 
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tains high intarest in die task is more cohesive, more skilled in group 
work and more ego-involved in the test because of its being a "group 
project" than is the group whose interest is solely dependent upon im- 
mediate personal participation in the task. 



RSSR OBSERVATION SHEET 



Obyervor 
= Date 



TiiChcr 



School 



GratJa 



"in ir sup 



PLANNINO STAGE 



1* Group Participation 



Individual 




































Buzz Groups 




































Chorus \ 




































Moiae_ H 





































Comrainleatisn Patttrn 



Suggetta Dlsorits Idaa 




































Hepeata Idea 




































Evaluatsa Idea 




































Ij^roves Idea 





































Gona true t ion Authority 




































= SuD§rvisory Authority, 


































. S^imultaneoua Group a 




































Sequential Groups , 


































Problem Groups 




































Non-problem Oroupi 




































All at~ onoo 




































Samo as last tiini 


































Qroup plsqipline 




































Change ^ Personnel 





































Prods group 




































Su^gasts vote 




































Reirinds group 






— 




















^ 










SumiMri^es ideas 




































Gondyots vote or eleetlon 





































^Aotlvo Oono€nBu3 




































"ActiveVoto = 




































paisivesVotg 




































passive Concensus 





































6, Deserlptlen ef planj 



Tina Plarinlng_ 



m 



80 



1954 INVITATIONAL CONFERENCE 



RSSR OPFIRATinri 



to 



Approach 

1. Plan put Inte effect omoothly and witheut hesi'tatlcn 
i! Plan put into effect immnd lately in a wild excitod rush 
3* Plan put Inte ©fftct after sdm^ heiitatlon 
U* Plan braaka down imnedlotely _ , 



The plan 



In OppratiPn^ ~_ _ 
~ Ohengad^lneff ic_ignt 




Tins 
Ns, of Errors 



^1 



. <^ TESTING JPROBLENfS 85 

f ■ : ^. \ . ■ 

Descriptipn of 'Grpup 'Characteristics 

JOHN Kk HEMPHILL ' . 



The ^tudy of social groups and the behayipr of group meinbers is 
rapidly ^^'oming an active field of eri4©a'^r for psychdlogists. Both 
ernpirica^esearch and theory building are receiving attention. How- 
evfcr, if the scientific study of social groups is to continue to be vigorous 
6 ana effective, one area of problems that has been largely- neglected 
must be given increased attention. It is imperative that, a taxonomy of 
group characteristics bQ developed in a form that permits accurate 
measurement. 

Research work on problems of group taxononiy may be less ex- 
citing than invest igatirig the intrrguing_proc€?sses of group function ingj 
but if the many discrete research firidings that are accumulating in the 
litefature are to be brought into some order or relationship, %ve must 
use common concepts for designating major differences among groups. 
If we cbuld specify the characteristics of different samples of groups 
by using more comparable terms tlum are now.availablei match eonfu^ 
sion that exists at present in group research would disappQar' Questions 
V about seerningly contradictory m^earch finding^and about thfe geher= 
alizability^of' the results of hilmratory re^areh would be inore^us^' 
ceptible to answer. . 

This paper reports the development and use of a tentative set of" 
dimensions devised for use in describing group characteristics, . ^> 

Development of the Group Dimmsionff Description Questionnaire 

The following four cnteria served as guides in the selection of 
characteristics to compose the set of group dimensions: (a) each char- 
acteristic should be rricaningful within a\sociological :or psychologfbal 
f rameworkj ^ ( b ) each characterigtic should be conceived as a 'nn=. 
tinuum Yarying from the lowest t^lK^lj^hest degreej (c) each char- 
acteristic^ihould refer to a relat&^^i^Jf^lar rather than a molecular 
property of a group; (d) each; i^^i^^isjic should be relatively or- 
thogonal or independent of all drffcr:Mamcteri^^^ in the descriptive 
system. ' \ - 

As a first step, and after a thorough review of relevant liters tur'e iin 
social psychology and s6ciology, fourteen group dimensions were 
selected and defined as follows:^ Hi* 



* A more , complete definition of each^imension and a detailed actount of the 
dovelopment of thfs instrument is given by Hemphill and Westio, (3) 



86 l9S4 lNVlTATlpNAL CONFERENCE 

'l. Autonmnyjs^ degtm to whidi a group functions Indepehd^tly 
' of ofter pdups and occupies an mdependent position 4n. society. 
'2. Contrd is fli#dfgi«6 to wUth a group regulates tfi#tehavior of 

indlvlduak while ti^ are functioning as group m ^ 
3. FteriWftt!/' Is^^ degree to which a groups activities are marked 

by informal procedures raaier tiian by adherence to estabUihed 



4. 'Hgdonte rong j» fl\e degre^ o which group^m embership is accom- 
panIe4byapneraifeelln|ft^easanbiessoi^eeablen 

5. Homog§ne^ is tiie degree to which members of a group are simi- 
' lar witii respect to socially relevant 'characteristics. 

6. Intimacy is ttie de^ee to which members of a group are mutually 
' acquainted witii One anottier arid are, familiar with the most per- 
sonal dietalls of one another's lives. • 

• 7. Ftff^poWohjs the degree to whidi members of a group apply 
time and effort to group activities. • 

8, Permeabmty is the degree to which a grdup permits ready access 
to Ynembershlp. ' * , 

9. Pdkhmtion is the degree to which a group is oriented and works 
to^d a-^Singie goal which Is clear aijd speciflo to all memljers: 

10. Potency is Ae degree to which a group hai primary significance 
' for its members, '^i 

11. Sfefi Is the numbe* of members regarded as being .in the group. 

12. StoMitj/ is'the degree to which a group persists over a period of 
time with essentially the same characttirist'lqs. 

13. SiraHficmdn Is the degree to which a group^fiBrs its ^embers 
intoi status hiprarohles. 

14. Viscftfrty is the degree to whi^h members of a grpti^ function as - 

. a unit. ' , • 

As the second step In the development of the instrument for measur- 
ing group diinensions, over IQDO itemi,/each referring to some specific 
characteristic of relatioriships am&ig%oup members 6r to ^ome man- 
ner of group functioriing were aii«(tfcled. These items were secured 
(a) by administration of open-ended questiops about the nature of 
their groups to membera of over 500 groups (2) and (b) by selecting 
excerpts from sociological, psychologi5|l, anf literary writings in 
flilch groups were described, _ ' // 

Data for an index of**'homogeneity .of plncement" were secured for 

each item as the third step, ' i ' 

This index was adopted to. give a .single numerical, evaluation of 
eaA item with respect to Its homogenplty, not In the usual sense of 
inter-item vari^ce. but in relation^ to the other thirteen dimensions 
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ioiight fo find items tibat were 
ensions al|p clearly irrele- 
f five Judg& sorted aHi€^^e 



composin^^a dasc 
clearly ralevant to one of 
vapt to the remaitaing fliirte- 

itenii fourtee^tiAes. Mter c^^P^Iy reftding th^ daJBnitton of one of 
the fourteen ^oi^ characteristics the Judges placi/^jeach itepianto dni 
of categories. One ^|egory corisf|ted of Jhpse ite^^considered 
to apply direedy to^the db^aracteristic.^A second category included 
■ tliQS6 items about wnich tBe judges^^e undScided* JTie' &frd cate- 
^goiy^^^ for iteins oonsjdered irrelevant or inappH^Jbl^tq^J^e char- 
J/ai^erisWc. Upon the completion of one sort^ thcfit^s were shuffled 
ari^ the same procedure was repeated on the next dimension, j^gree- 
ment among Judges that^, iteni applied to a dimensiofi and agree- 
ment that it did not apply tb othe^ dimension^ in the system are given 
approximately equal weight in the value of the index.^ * 

The fourth step in tfie development of measures of the group char- 
acteristics consisted . of a conventional internal consistency item anajiy- 
sis* For diis purpose a preUminary questionnaire composed of the 3S0 
items Aat had highest index valpes was administered to 200 individuals 
who were mfembers of 35 different groups. Thpse respoftents' indi * 
cated how well each item described their group by selecting one *of 
five choicest Deflnitely True; Mostly True; Undecidedi Mostly False; 
or Definitely False. 

As a furtl^r refih§mentj a ^smaller sample of the respondents who 
completed the ^0 item questionnairef^ere interviewed and questioped 
about why they had selected speeific ehoiees to, specific iteins. Ambi- 
guities in reasons for selecting responses were noted and then each 
iteni was edited ^o eliminate sources of ambiguityj or removed from 
further consideration. * . » 
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The find instrument entitled "Groip Dimeniions Descriptions" is 
wmposed erf 150 statement abdut groups each assipied to one ot 
aurt^ dim«isIoiis.» A rMpondent's description of his group w sum- 
marized by thirteen povp dimensi^ sc&res* Each score "the sum . 
S^e weiAts given Voices -Deffitely True' through Definitely 
^Ise.' for die itims wsign^ to each particular dimension. , 

^^^^^^^ X \: 

The final form of the ques«onnyre has been used in severalempiri- 
cal studies stace its deVelopm%t 1#1950. Group descripUons haVe been 
seeur^ for office and shop groUps in industy (1). departtnents of m- 
itaiction in universities and coyeges (5), high school and elemratery 
^achiiitt staffs of 28 Ohio publib school systems (8), v|rious mditary 
C7).-wligious organizaUons (6), sports teams, informal, wml 
arodis (4) etc. We have had opportunity to examine som6 ot Jhe 
>o^BS of the dimensions scores that are available from :th|se 

'*ftAaps^e most important question that ^an be asked about^e 
group dimensions scores is whether or not different member* of *e 
Sme soup agree about the characteristics of their group. Unless sorne 
aw^ent can he shown there would be cause to .question seriously 
the^tence of such an entity as a group. W#have examined toe sets 
of data where it was possiBle to compare the yarlance of group 
dimensions scores for ^e descrigtions supplied by members jOt the 
-same groups witii estimates of the- variance of tiiese scorers based on 
■ mean scores for different groups. Signlflcant F ratios have been found 
for each of the diirteen dimensions but not for all dimensions in al 
tfiree studies. In a study of eight groups that were desdfibed by a total 
of 65 of their members, the extent of agreement among de^criphor.- 
kppUed by^the members of the same grbups, "Skpressed as unbias. 
co^elation ratios, ranged from .47 for the dimension folarization to .74 
for the dimension PermeabiUty. In a second study involving 134 faculty 
members and their respectlvf 20 departments, significant betwee|i. 
Br8up variance'was fonnd for the dimensions Flexlblh^ Hedomc Tone, 
Intimacy, Particlpatf on, Polarization, Stability, Strati&ation. and Vis- 
cidity In a-ttiird study tiie staffs of 28 elementary public schools were 
shown to diffe? significantly on Autbnomy, Flexibility, Homogeneity, 
Intimacy, StabUIty, and Viscidity. It appears evident. that die descrip- 
tlons suppUed'%y members of their groups with the use of the Group 

"•SizB wai dropped from the final fonn of the rfufflUonnaire not becauso it was 
CO JttroTanS^ant dimension but because tTiere appeared to be more direct 
meAodsaf obtalnJoj^lnforinftticai. 
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Dbntnsiona .DsioripttOD Qu^ti^ptidft proyide diffeffntiations Md^g 

Soma ^^Q^! ^ Cba mUdJ^ of such differentiations hai also bean 
' ol^^^^, Fot egmmple, d^od^tioni of a militety^ttainlrig department 
^ROTG) wercf oonfrast^ wttt* d^oriptions of a committea from a 
college of ^u^tidn and flie a^^cted laifge dlfference^^ong di-^ 
mensions of Contol, Fiuticipation» apd Stratiflcation clearly ^aerged, 
■ Not' all of variwca among r^pondente who desmb^ to smia 
group is to be a^^uted to errc^ of m^surement The dinieniion scorai 
h^^^ bras, shown be rdatad td. di^aoteristios of the responden]^ 
rti^is m poup. For exmple, Ba study of Ae dcpartmente of a 
lib^al arts colleg© it wai discovered that t^ose memberi of die faculty 
jiyhi possessed greater status, i*e* held the rank of Assistant ftofessor 
I- or Detter, described their deparbnents witti lower scores on the dimen- 
//rlion Contol than did oAer mwibOT of &eir d^^taen^ who were 
AV'Targely Insfafuctori. In Ae same study it wblu also found Aat diose in= 
it ^ dividual mambers of the faculty who e?^resied above average satisfac- 
Uon wUi'^ their jobs tended to view; tiieir d^art&ents as W^er on 
_^ the dimensions Hedonic Tmej Participation, Polarisation'^ Potency, 
and Vfacldityj but as lower on die dimensip^ Stratiflcation. In if study 
involving aircrews^ which used a modified form of this questionnaire^ 
Rush\(7) fo^d that Aircraft •Commanders dtfFar from dttif r orew 
mraibers^n the*way tihey describe tiieir, crews. No study has been 
made r^d^Ar§^ measures of die .personaUty variables of respondents 
to thefr descriptions of their poup, but sucR* relations might be ex- 
pected* TOus w^ ses^at die ^ou^ dimensipns ' scores are sensitive 
both to conimonly^ seen .group characteMstics and to die respondent's 
partimilar relationsw^jD^ls poup. ^; > / 
^ Considerable efort in die develppm^t of the Group Dimensions 
Descriptions questionnaire was emended lb atteniipting toi secure inde- 
pendent dimension scores. Jn order to exaiiilnei^'' exWnt o^ overlap 
amop^^Aa dimensiQtf scores empirically we hayq com^ted intarcor- 
, rektiona^or ttree sets of data; a sample of 100 deiCfiptio^s of miscal- 
JL laW^is gfoupSj a sample of 136 descriptions of 2S deparbnents of a 
U^^al ^rtl .^^ege^ and a sample of 315 descriptions of die staffs bf 
28 Ohio puHUc ^c^oolsystems* In general the COTfal^tons among the 
dimtwions fflfo' quite smidl— approxiD^ately tiirea quarters of them be- 
^ low ,3 in ma^itude. Certain dimensions are fcequendy f^md to be 
coirelat^ more hi^y widi each odier. For exampM the relationship 
batw^n Hedonic Tone and Viscidity was fouAd to ha ,33| ,64 and ,57 
" fpr ^e three samples respectively; between Partieipatiqn apd Polariza- 
tion; ,09; i34 and *47; and between Stratiflcation and AttttonomyJ -—^B4; 
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* - .31 and - .24* Tlie'size of ffi© corralationi between speclfla X 
tnensions appear to be seniltiv© to to IniUtutional setting from wWcl^ 
the ^upi are saleeted. E^rience aqpumulated to date has pro- , 
vlded no dear-cut evidencis for Ae abwdonment of any one of the 
dtaeniioni tiiey are to be eomid^ed' applicMe to aU varieties of. 
poups. However, it does appear tiiat if to ^m^^ 
only indifferenees among a relatiyely homogeneous sample ^©f groups, 
certain of to dimensions might b# pe^^nt example, m 
adaptog to insSument foTuiS^^ 

meosions of Si?e, Autonomy, and PermeabiUt^^Qiuse to assumf^ 
tf on of yariattqe on these dimehsidns a^peared^rtpKtehed. 

Finallyj.to problem of estabMsKlng normative dWa for tose group 
dimensions htfs been given dttentioh. I frankly do not Imow how to 
establish defensible noims for group characteristios. The problem ap- 
paars to be one of lifting yourseU by your bootstrap^ Certainty, secur^ ^ 
Ing representative samples of individuals would no^insure a repre- 
sentative sampling of toir groups^ since individuals have numerous ^ 
and complicated multiple group rflebbOTship. 



jjie development of a tentative sef^bf dimensions ot,group chgrac^ 
teristics has been described. Some of the ^perienqe that vfe have had 
wife to use of the dimensions has been cited briefly, Mu^ remims 
t^^ejone. We nee^ defensible noKns.^We need to reconsictor^ to 
selectio^ of ^oup chdracteristia| Perhaps the list of dinlens^ fhould 
bi enlarged, or perfiaps it can Se reduced. Hfct unf ess we lolve some 
of tose elementary pr^lems of describing the M»io entity of ^eardh 
oil group^ e^onf^sion will reigi| ^ '^b ^ 
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PARTICIPAfIri 



William Coffman, Richard S. Crutchfield^ 
' M. pAvisoFr, LiNDsiY R, HARMONj Jom^ K,JBe5<rhilj^ I 
ELLidrr McGiWiEij Emm^ 

Db* !bAVii©N: I ^av© a question for Dr. HeijfiphilL Say you ha,^ 
groupj and it has dm^nsions on your test p^er l>ere to tiie right bp 
d^MbutiaQ. You have anptiier one over hare ta^a,Jeffr a^ttfev^ 
tion, No#^ how do I laiow whether or j^^if l traded pcS>ple^ound,i ] 
tii^ womd not comcideP If it poislUe^|Hj|j^^ pll^ their own 

frame of jbehavior and therefore^ might irlSRftftssible ^ really com- J 
pare: poups? % ' 

* D«* HEMpmLL: You are asking a-^estiorf 
I can anijs^er* I wotJd suggest one things ho 
such a problem by using tiie questionnaire fl>^1 
hand, E 'would not attempt to answer the 
change^^ffie^ group when we change some or aj 
would Iw^e to know much more about the gro 
make^tf^uess, I think you need a tool or jnst 
cussioil 0ue4flonnaire. ^ ^-jy^ 

Db* Lqrge* J was hoping^at the other queitiqftiATOyd^ ^sked^^ 

1^ ^oups to whi^ Kc belSn|ed,^cQul4 
f'^e^group of #h^dte;ijte ifi^ii^meinberj: 
i grQuipmember^pife , ; : / 
ive tp give th©"iiime ;|(nsW^ 
Buld stiidy the©obleii^V^ 
to^/ J shoul(fflifeit^iN|-empha^ 




p not know diat ': 
ight approach ^ 
esoibed. Off 
, VhatheJ. we 
ierions m it ( ' 
if^ cduld ev^n 
^fieSroujp Dis^ 



J£ Ae individiifi could ideni 
you Aen spedfy the groupni 
and if 'there be homogeneity 
Db. IbMPHiix^ I t^fink i 
your question. Professor Lprj 
know .what tte answer might to 




size, howevCT, that we are'quit^^oi^P^eii that we a]|i jiudying;^qme- ^ 
tiling involving groupness WnJ^re^^||jl^fo^j^ 
of a group give d^criptions,^f^^eJf ^^ff^ dc 



^on^of otfier groi 
I Dr. WmANS 
would, like 
and |econd,;I 
ment on the 




er^ 
escrip- 

tw4j1i6lBfe;;^n^ First of aij 

^ to shp^^s the other^rdblems ghe irhentie 
PprfciaM^t if Professot Hemp^ilKwbulS coirf^; 
e betwf e^Jiis items per^&fbiUt^M^d' stabttityy 






^i^i All-told you^&e prbb^pis are aesimed so thittiiey 
.mu,e™..%giffly in order of difflQulty. In the first prdblem, the Ijouse ^ 
|r^^|inofice :ttat the colors are alt togeth^ i^rai^t^i and- / - 
Ih^he's^are'and Wangutar blodts v u ^ ^ 

'f ^In thi secbnd problem, the tri^ aie^eolors are saU s^arated, ^^ ^ 
th>/^ri^ and iquares are mixed, snd are tftus more; diflaciilt -^ , 

put together., '4 . , . ^ * t ''^S^-' 

,In the tod' goblem. the ;do g; both thiB cpias ajjyiape&'of m 
,bl<j0ks are intermixed, thus making, it the most dlffllulfy^l.;Oirto^^ , - • 
problem, by tte way, the children arc asked to-beA.flieir tiiperon ftf. 
' bridge problem. That is, they do not Jet fifteen. rtinute^to^^^^^ . 
' get oniy^the numWof minntei It^ 

' icfroe& them to evaiuatQ andiinp^^ » 

'•Dfl/ HEMpfcLLr I p^^ very; short deflnitiftns offtti|^ dimen-., , 
Stons. X thiilk a Uttle elabiraHoh 'wpuld makelcl^r ^e diffirenee be-^^ ■ 
. tween stability snd pnneabUity,^Bernieabilit^"refers to a qlferactor- 
istic of thb group Vrtiieh Is oftpn reflected in maq^erslHpieq^iti^nts, 
" or tiifc tfifflcUlty' Of gBtHng into ttie group— wl^'yofe hav^lfo do on be 
in : ord^ to;^iialify- f or roembiBrshiih StabUit)^ ^0^^?jP whMiSSr,%;> _ 
'group remairis the with 'respect to all signiflfcanfe cSaraGtOTStics 
: t««r a; p4ripd;br tlni a stable Jreup .iff 19^ misH , 

/b^^ily llkdStwasinl899. > ,'; ■ ' " * ■ — ' 

8WtBYf I should^ h^^^ ask Dr. Damrin v^th3r shebjans to ^ 
,:;^xtriii he^ expert to include verbftl, or does she plfen to IWy W*' ^ 
, : whatiWy haSi^ I am 

/ ■^^^d'^^I9IN^ ;^ ttie prpsehl timo, we do not plan to|xtS^ th^n^ ; 
s swuetipn a set of tests tlratiare tawo v^alw 

^ Ortl^'chiW reason: ifr that vterbftt tests depend rather h^Y%..^#W'" 
''■JcitfcdrnJid-NVeawi^^ ' 
" ^pos^ibl6' Tfosep^ ai-e suifflciontly pim^ that classes W < 

Veiy' dun children <lo tli(nn. In fact, we ha%j|icpesd^ adnynis- 
:jtoeiitheihtoclasseswith|in,avw^ |* " 



appropris 

m J haVfe given .^e test to my rescarcH boarci, aii or wr 
tifid m gerteral the obtained resuUo^ere cftiite satisfacd ^ 
; qitf present SGOring shtem. Jt is 0^= feeling, the^tae,.vthjB to^ 
■ & ;area of veAarprobjem solving would sarve ^narflylb a4d un^ 
. ifebessary complexity, and the test as it now stands, ftl^ad^as so ^ 
much comple?rity that it Is extrpnjely difflcult to ^core. ^ ? ' &, 
■ . Db. Hv^bmon! I should like to ask fir. Crutchfleld^^ & any ^ 
. fommtion on the validity of these conformity scores against cHte^, 
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^ of fiM^]^ca to die iqdivlduaf s adjuatment on t^a job^ or anytiUng 

> ■ 'Ow^ Giuroifflmaiir dp have som^ ©xtOTrf caiterit of parfiorai- 
aboa in the s^dy €i{ prof^Monal men^ HeJ^i lerb order opirrfiatiqns 
^'^fiS^t^^ not seeto Vejfy promising. Thfri are, how- 

brf gu^sttms of ^ttfc^tiier &e critarfa are adequate and how they 
^dmd W pi]^ct@^j6 rebte f o dohf oflo^ can really $ay 

Jko^Jta^a^^§$:^ to,.pp|nh cl j WK re- 

TpB* CbFMANi I should l&e to ask Dr. Dani^ a question* She has 
I nfi^e the stat^eht in die general presentation tiiat she has no data 
' o1a?^v^idity or on Telitfbflity and did not'add Ae statamOTt **of the tra- 
ditional sort.'* There are three qu^tions involv^ here: one is* does 
tt'^'?tha behavior. of tiie dasses on tiiis test relate to any other evidence of 
= what the ^up probl^-soliing abiUty of the da^s is? Second, does the 
test differentiate among class6sP And third, can scorers see the same 
thing? Do you have any e\ddance on .ttiose questionsP 

Dr, Dxi^Nr My statemeht tiiafc^I have no data of a fonnaLtype 
rrferred to tiie fact that 1 have no tables of coirelation coefflcients to 
showyoul 

In Mswer to the first qwestionj the prettd study on 
in Trenton, New Jerseyr The elementery school supervijpor tiiere loiew 
her teachers very w#ll and selected fpr us, but unknown to us, pairs of 
teachers, lliat is, from any partfoular school would pick one teacher 
who she thought was tiying to get across in her classroom the tilings 
which tiie Rusiell Sage test is concerned withj and for the other she 
would select a more foftnal type teacher. In every instance but three, 
tiia tests discriminated these two types of teachers/ In otiier words, in 
the mora fonnal class tiiere would tend to be very low group partici- 
pation; in the other, greater participation. In one class, student leaders 
would tend to emerge; in the more fonnal class this would not 
occur, and so on. Therefore, we have tiiose kinds of evidences about 
tiie validity of the tests* 

To the question of does tiie test discriminate between classes, the 
answer again is yes, Rreliminaiy sealing of Ae scores tiiat I have de- 
scribed to you reveals that classes differ very markedly from one to 
anotiier, and yet tiie same class is relatively consistent in its behavior 
througtout the tiiree problems. This gives us one type of evidence 
; about test reliability* . ^ 

To the tiilrd question, d0 scorers see tiie same thing when observe 
ing diildren on tiie test, I can only answer tiiat I hope they do. I have 
. just finished training two teams of examiners in New^ork For tiie 
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planning sfragft we $e€ni to be getting reliability thaV I would roughly 
eatmatf is about ,9* This is on tiie basis of one week^training. On the 
operation^ stage I wn afraid tiie reliability is aboift '^^This is because 
factors such as the "psychological tone quality" of a groi^ of cWldren 
require much mOTt actual experience to evaluate accurately ^an do 
factop sudi as *'annount of group participation/' . ^ j^i*' 

My present feeUng is Aat the data for the^pationi stage are not: 
3^t usable baMUSerfthfeia^ ; ^ ^ _:_ . . ..^ 

Dr, McGiNNffli^ Dr* Crutchfieldj I was interested in yo^description 
*of some of flie parameter that seemed to influence conforaiity be- 
havior, pwticularly those measures of personality characteristics, and it 
sounded as thou^ the nonconfonnists had' all the desirable persdnali- 
tie?s vilhile the oft'ers had the undesirability traits. If Ais is frue, I 
wonder whether you could say a little bit more about some of the , in- 
sWmehts ttat you used to' arrive at these conplusi^ 

D»* CBmcmmi^i 1^ should emphasize that the particular .way the 
studies h%ve been conducted has not been such as to measure* extremes 
' of nonconformity, that is, negative suggestibility effects, so to speak. 
This could easily be done. There is some incidental evidence in the re- 
suits which would.iead one to suppose that overinsistence on independ- 
ence may at timeJ be associated with undesirable perswiality features. 
But this has not yet been fully investigated. 

Coming to the main point of your question, die instruments used in 
the personality assessment are very varied. The assessment of die 
professional men and of the mature women lasted three full days, and 
involved'such proeedures as standard tests, specially designed person- 
ality inventories, group interaction procedures of various kinds, inter- 
views, projective techniques, physiological and medical measures, and 
so on; in short, the usual range of things found in large-scale assess- 
ment The coffelations I was reporting are drawn from all of these. 

One rppresentative finding is from the Terrean Concept Mastery 
Test, a newly devised instrument measuring superior intellectual per- 
fonnance, wKich correlates approximately .50 with indep^dence. 

Some of the scales of the MMPI and similar personality inventories 
were also used. Anodier very important source of inf ormation came 
from Q-sort descriptions of the assesses made by the assessment staff 
in complete ignorance of their performance in conformity proce- 
dure. Here diere were found quite stoiKit% relationrfil|s along the lines 
I outlined. , ' ^ , 

Finally, since these days the famous or infamous F-sck|j studies of 
^ personality are much in the litera^^re, I should r^rt ai^ppropriate 
finding tfiere, th^ correlation with conformity is order of .40, 
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pAptts, by dmiffcLg malca a b^uttful imity M ^ii bupoitaiit areai T^ey 
give a s^a^uj attadjgt to get at a conc^t of nature of tibe ^up, 

. tiba pioisibiUty Att enablt worfc^^ to mterpret flie da^ae 

to publ^ed evidta^ about grouj^ i^ally repreaente Intepated 
and functionji&g gcoupi. Ferbaps I am too diiturbad by ad hoc ^oups 
OMted in ^asaaa by insbuotof a smymg^ ^Ilia next five bidivMuals will 
^lititutt group 1 an grou p 2. ato p for p u^p c^^ of 

measuring group p^omance or group produotivity* TTiey show thatv 
. in s(Ma proems bfasarv^ati^ it is quite clear tbat^oup bebavior: js 
jrelated to the fact Aat some members may not work at all. 

And Ae papers show interesting dual relationships in attempt^ to } 
evaluate group probess as well as tiie group product He complexity 

, of ralatipnships e^^bited ara: (1) batweqn kinds groupSp be- 
^tweaia ^I^di of membari J (3) the natura of th© procasses'Telafed to 
gif^gs and mambars, and (4) dia quality of the product^ whather it 
be sTsolutionj an action program, ate. ^ 
0 A large numbar of theoretical issues come to the fore, because;, as 
far as I oui tell, the evidence about tile relatiohship between group 
process and ^dup pFo4uct is Just emerging, More observations and 
data are needed, not only in tiie area of action^ but also in the area of 
tihe qu Jity of the group solutions as opposed to individual solutions. 
The distinction betwen tiie productivity of individuals and of groups 
must be made. 
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When your chairman invited m© to address you here today, he aug- 
g^t^ tot I need not necOTarfy talk about te^i^ fMfoblwjs^ but 
^t I mig^t disquis^wiytiiing I Aou^t relevant to die pu^oses of ' 
your conference* I flierefore, deal wifli some problemi wUdi 

hayl b^n forced upon us by the events of &e la^ fifty yem. Itey 
^6 especlaUy Adtal to aU of us engaged in tiie study^of human beha\^or. 
Th© solution of Aese problems will most assuredly ect our well- 
being. Failure to solve tiiese problems arighf is ^tain to affect Vol- 
canity 4e cultur©^ and welfare not only of oursdves but of mmy 
generations to come* 4 ; 

The problems to which I refw have become partaeularly acute in ^ 
our time because they arise out of tiie wide, y|Lwning gap that erists 
tdday babveentitie tremendous advance in science, paWcUlarly ipph*ed 
science/ and ffie appalling backwardness still manifest in human rela- 
tions* Our advance in scienlfflc knowledge has bfeeri: so rapid that tih© 
dr^ms of Jides Veme have seemingly come true, but our backward- 
ness in himiafie bdiavior, in dealings be^een individuals, between 
p-oups, between peoples and nations, has been so retarded tiiat it is 
open to question whether tiiere has been any progress at . all to 
tibe general spiritud level of human relations since die Greeks took. 
Troy with dieir wooden horse* ' 

First dien, let us look for a moment at the growtii of science. Interest 
^ in scientific mediod began to germinate, to be sure^ as far back as Aris= 
: totle, but more especially^ three centuries ;pgo when Francis Bacon % 
wote his Admmmnmt of L&apiing (1605). But it is only during 4© 
past fifty yean that the great burst in the growtii of science has oc- 
mmed, I tlnfak it is fair to say tiiat more advance m scientiflc taowl-' 
edge has occurred during die last fifty years dian in all previous cen- 
turies combine^. We are all aware of this in a general way, particu- 



larly tiim^ of us Who^ persohal recoUe^ons go back to fte turn of 
the o&^tQi^. Wa Imow from ogr own obiervation the fantastic Pangea 
m £be aumhw of J^lephones, automobile, machines, and gadgeti of 
all tods. But letme dpal briifly, diough in reasonably specific detail, 
a few fields rth^ serve to dramadza ttis ift^ecedented 

rowfli. f r ■ / 

Tdca, for ^tm^e, tkvel and transj^rtation, \^en George Wash- 
ington was Preside^ ^ United States, it took longer to travel 
from his home at Mount Vernon on die Potoma0to New York tihan it 
takes today to fly from New York tp Bombay, If we visualtoe die eardi 
in 1800 as a globe eight feet in cliMneter, tliis glob© in terais of travel 
time had shrunk by 1900 to die sl^ of a basketball and by 1984 to die 
size of a golf bUl, . . ^ 

' This phenomenar advanae in scientific kiow-how is fundamen^y 
assodatsd with the d,evelopm#nt and use of energy, Man has been able 
to inCTease his productivity, not by inCTeasing strengdi of his arm 
and back, but by the use and control of energy in die form of tools 
and machines, i^ound 1800 steam engines were just coming ii^to use. 
By 1850 die United States had two million horse power, or one-tendi 
oif one horse power per oapita. By 1900, it had risen to one^half a 
horse power per capita and as of today die total energy in all prune 
mover engines in fte United States is over Six billion horse po^er, or 
about 39 horse power per capita. (Estimated by John A, Waring, Jr., 
Bteelways Magmine, August 1954.) This is eighty times as much me- 
chanical power as was avallaHle only fifty years ago. Productivity per 
man hour in manuf aetiiring industries^as iriereased fivefold since WOO. 

This enormous advance in technology has been stimurated by, and 
made possible byt^ die manitold increase in technical education. Around 
1900 there were 1,150 electrical engineers. Today diere are 46,000, a 
forty fgld increase. In all flelds of enginefeiig combined, ftere were at 
the turn of the century about 13,000 members in die various profes- 
sional engineering societies. Today there are 170,000, or thirteen times 
as many. * ^ , 

But this is not all of die broad picture of rapid change. Take die 
study of man himself, his characteristics, behavior, and motivations* 

/Here also in die last fifty years, a fremendous acceleration in scientific 
research and Imowledge concerning human behavior and human rela- 
tions has taken place. To cite merely one example, around 1900 tiiere 
were only about 123 members in die American Psychological Associa- 

^tion* Today diereare moriitiian 12,0OT, a growdi in technical frained 
personnel excewbg even the relative gain in members of professional 
engineering sod^fes. Add to diat similar^growth in number of toined 
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eoonomiatk spoi^ogisti, pd^ scieptlsts, and historians and you 
Jiave a rfiy imprdsive airoy^ professional talent devoted to human 
^-^ties ]kn4,r©i|tiong, yyhere^ fifty wars ago ^e picture was largely v - 
■ one of soUtwy^pioneers. y ^ v . 

Take Mother n^ct, ^annels of communication, messages mnd ma- 
terials for influe^ng p06ple*s^behayior have increased at an miprece- 
dented rate botih in liunAers a^d in speed of transmission* Today there 
are In &e United Stetes one and one-fourfli copies of newspapers per 
. day per family, two and one-half copies of magazines per week per 
family, two radios .per family, and one television receiver per two 

families^ . 
Mass communication haf so speeded up/ that it is almost instanta- ^ j 

neous. An taportant event occurring in Washington at ^breakfast tinie j 
^ will be toown in all importtuat capitals of the world before your toast , 

has cooled off. By confraitp^ when Lewis and Clark arrived, at St. Louis 
- in December*1803 to statt Uieir, westward ejfploration, the Spaniteh 

ooni^andant stationed at St Louis refused to let them cross Ae Missis^ ^ 
; sippi because he had not yet redeived official word that the territory ^ 
^ west of ihe Mississippi had been ceded to France and sold to the 

United States months bePor^f 
Look at still another area— eduWion. School enroUme 

an immense upward surge. The number of pupils arid the number of 
years in school have increased by leaps and bounds in the last fifty 
years. In 1900, fte total enroUment Jn our high schools was 700,000. 
Today it is 7,000,000. Around 1900 the number enrolled in our colleges 
and univeSities was 1 HOOO. Today it is 2,500,000. In other words, high 
school enrollment has ihcreped terfold and college enrollment more ^ 
^ than twentyfold. Put still another way, 80% of ' our yqung 'people of . , 
^ high school a^ are enrolled in our high schools and 30% of'C6llege\ 
age are enrdled in pur colleges. Hence, as of now, of all adults'2S years ^ 
or older, 60% have attended high school of whom one-fpurth have also^ 
attended college. ^ ^ ^ 

: . Now iet us take a searching Ipok at what happened in the man^ 
■ '^gement of human affairs. B^i there been any improvement at all in 
^ the spiritual level of society in the jpfty years during which these ma^ 
veloits advances'in th%physicaf a^|^cultural standards ha\% occurred^ 
* or even during; the three centurifs since the* day? of Francis . Badoh^^ , 
' m a ylp\i will,:§inde the days of ^i^totle, Hoses and Confucius? JHero 
we are^tt a field that is without a generally accepted measure of value. ^ 
But speaTcing in all humility, I A onj doubt that we have made any real - 
progress. I doubt ^hether we mve Kamed to live more wisely. We cer^ 
tainly have not ^ferie^ to livp jn peace. Oh the contrary, j^ars have not 
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shed'jiifr0quenqy and have< become itiore brutal and destructive 
?|ikhah''^^ cost the United States alo^p 30 billion 

.^^^^ 380 billion dollars. But far worse, 

; : ' 6r'^ of human life. During {tlie Thirty Years 

. Wax: thiree centuries ago, one-third of the populatibn of Europe died. 
; During the first' and second WoErld Wars lasting nine years, twenty- 
\ five milliompeopl^ were killed. It may well be that more people have 

tt^ Qf jast fift y y*^ir tjjli? ,in all preViou 
mstorj^, and\,these a^e the ^fty years duririg%hich h^fi occurred our 
1 boast^;rise in, teclu|()logy and Icnpw-how, botji in the physical sciences 
;aS well as m the human behavior sciences— in schooling and in the,: 
^ mbans of rims's communication for influehciilg men's lafnds. Can it be 
that we have nSt^realized that with incrliasp in technical knowledge 
there must be a corresponding increase in "rpsponsiBle humane be^ 
' halrfor if wholesale brutalfzation, pay even universal ^ to 
fee avoided?' ' . — ^ . 

TuJ^o the behavior of persons as individuals in their faiAily and 
community rrfatiom. Acco)rdihjg to United Statei govemnienf statis- 
tics, th^re was,«i 1900^ one divorce ♦ta thirteen marriages. Tpday there , 
is, one. divorce to fcur marria'ges, a ratio thfee times as -HigH today as 
fifty years ago. According to F. 'B. I. records, oflFenses inbluding murder,.; 
nriariSlaughter, rape, robbery, assault, burglary, larceny, ari^ auto theft , 
increased 37% in the thirteen y^'ar^ from 1939 to 1952. 

Why is there so^ wide a gap bet>yeejft what we kno\y and what We 
do with v^hat we laiow, between knowledge and th^ wisdom of hovy^to 
. use that knawiedge? There Are at least threp reasQ^The first Js that 
' rtiany people do not recognize that vkft(||rt)'&^ yse are two dif- 

fe'ent tilings. Knowledge as such dpfes not guarantee right use, in fact 
il does not necessanly assure use any caii de- 

tettnine use. The iavention of a. method of Shaking fire was a landmark 
/ in the march o| man, but knowing how to make fire did not determine - 
•'the purpose for which the.fire was to be used. It could tensed to' warm 
hut tod cook the food, or it could be used t^ burn down theheigh- 
,^rs iiut and destroy his crops. Wliiether nuclear energy .will be used 
rfor man'$ bepfefit or for his destruction will depend on purpdse. Science 
provide? technology but not^teleologv. " 

The ^second reasoA^ is a by-proJIct of .the first Technological ad- 
yimce makes comfphable, luxiirif^s il^ng an easy choice./This tends 
to ^sh aside long range consideration of wise l^iving. flel so con^ 
fortable, so luxuriouis, so smart that wig falsely^feel self-sufficient, su-V, 
pierior and trouble free. As necessity is the mother of invention, so 
trouble is sometimes the mother of humility. 
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.Tha/tiiird reason is that Imowledge generates power, and power 
over tiie forces of nature, and particularly over tiie behav^r of man, 
whete the appetite for more pov^er, foregoing rough shod over the sen^^ 
sibilities and rfghti of otfiers. Unless: povirer is accompanied ^pf^ ' 
increase of sensitivity to the common vmBLl, it ends in abuse, oppr^sioA 

..^idJto^ty^ LMd^^te^ "Power corrupts, absolute power 

corrupts absolutely/' u- 

The net effects of our fabi^lous advance in science are three things r 
we Wprk less, about half as many hour^ as foi^erly; we live longe^, 
about 70 years against an average of 25 years; and we live more 
iFortably. Quaere] however/ whether we live more wisely and more 
happily, We woiAip Ae god of science in die laboratory but unfortu- 
nately neglect ^^goddess of wisdom in tiie temple. As an interesting 
side-light, consider Ae case of the five employees, all foremen, sent 
in 1953 by the V, Nederlandsche LftioleunSabriek of Amsterdam 
to work at regular American wages three months in the Annsfa-ong 
Cork Company plants" in Pennsylvania. They came not to learn new 
skills or to obtain technical knowledge of linoleum production but 
solely to get tiie *'feer of America/These men were greatly impressed 
by the facilities, the ownership of automobiles, and particularly the 
amount diat they were able to earn and save during their short work- 
ing period. One man commented that he saved enough to buy a good 

_j^ed automobile and would have bought one if he could have driven, 
it all the way back to Amsterdam. He questioned, however, wheflier 
an American is any happier with his automobile than a Dutchman 
svith his bicycle. 

What remedy is there? How can advances in science be made to 
produce advances in wisdom? The remedy is not, I am sure, as,some^ 
one suggested, to halt research in order tQ give ourselves a chance to 
catch up with our knowledge, That would merely halt the progress of 
science widiout any gain whatever in wiser living. H, A. Overstteet 
has remarked, ''Better a dish of iHusion and a hearty appetite for life 
than a feast of reality and indigestion therewith." But this alternative 

, so vigorously expressed is by no means a necessary alternative. By all 
means let us continue research and more of it, but let us also search 
to find out what needs to be done to live niore wisely. Therefore, I wish 

' to suggest for. your consideration a three point plan of action to close 
the gap between the advance of science %nd our discouraging deMn=- 
quency in human relations. . 

Having recognized that knowledge of itself does not automaticallv 
lead to wise use, let us then first of all make it a specific goal to seeW 

. wise use of scientific knowledge. Psychological research has shownr 
tfiat trainings in precise, logical thinking with mathematical concepts 



q ^ 



^TESTING PROBLEMS ' 101 

^oas not automatically lead:,to mor^ acourate, logie^ diinldngm othar 
areai and witb pftar^day^and conceptf^^ 

It is essential, tiiCTefore, th^tiall people qf.goodwiH keep their goal 
of constructive use - of sgitace bright and shlhin| both for theinselves 
and for otiie^, i^oie ^vjdl evl| Jntent keep their riunds firmly ^ed on 
Jhw goiJsj^^B^^ likply to ggj^Qpfl passively 

Mdtiibut mudi thought about m|lr direction. The goal-cbnscious per- 
sonsj wheth^er Aeir goal bs gopjd fvll, have ttus dynamic advantage i , 
They know wKer© tiiey want to and they keep going. . 

What if wise use of toowledge alod how can we detennine whether 
a given use is wise? The dedpritj seif "respecting Jife, whatever else it 
may reqmrej has Ais one absolntely unfailing requirement rFreedo 
of cftoio#/ fif^dMi/of djoice in 4oit^t,.aftitudet speedis action and 
D^upation^ and witii it freedom to seek satisfying reward according 
, to efiFort, ability and achievementi^ In order to give proper re^d to 
the ri^ts and opportunities of others, in order tiiat everyone may have ^ 
maximum fireedom of choice in out complex w^ld, let us call it tiie 
optimum freedom of choice. ^ 

Hie second point of ftction that I submit for your consideration is 
that we must le^m and teach how to make sound value iudgments, Dis- 
^ coveries of science and uses of knowledge can and* must be evaluated 
by their contribution to the optimum lifej toward life at its best With 
its masmum freedom bf choice. Speciflcailyj how can we do this? Let 
me suggest diree criteria, ^ < ' 

1, The tes^ of cau^is and effect. Human behavior events are hnked 
to each otiber by cause and effect, They do not Just happen from no- 
where, TTiey arise out of previous events and actions combined with 
- human motivation. The first test, then, that I wish to propose for sound 
evaluation of ideas arid actions is to ask tihis question, Whaf will be the 
effect of the idea and of tfie proposed course of action based on it— 
not merely tiie bnmediate e£FeQt but especially the long range effect? 
One of the distinguishing characteristics of man is to be able to tiiink 
^ead, to project himself into the future* But so much of pur tiKinking 
is entirely in tenns of immediate effects when in fact the important" 
dynamic effects are tiie delayed, long range consequcncis* The imme- 
diate effects may be 'Weet as honey in thy mouth" but the 'delayed 
effects may be '^bitter in tiiy belly." 

Two tilings are essential in visualizing the chain of cause and effect? 
full information and objective interpretation. These are particularly 
urgent when dealing wltii current problems and events as they are 
reported tiirough tfie many channels witii all tiie screening and slant- 
mg albng fte way. As an editor said, he did not care' whetiier people 
read his editorials on the mastiiead pa^, he put his editorials into the 



102V^ ? 1,854 i^VlTATIb ' ; 

headlMes on tfie frontpage. To ttt the e^t^t to wWch ne^w toed: 
and screened, eompare the iame everiti aa ti^y are reported in di^- 
. ent ne\s^papers, magazines and over the air. Note espeQially the dif - 
. Ierence#n the headlines ani lead, paragraphs. You will §ee whatsis 
omitted/ what is Emphasized, and what is slahted. In a free socie^ 

cannq^tod^mustrotjarti^ 
sources a£d. then ask, Whrt seems, nearest the truth and the rightP 
2= ^Tte second cAterioh for making sound value judgments is. the 
tedof spMtua sur^d value. Wm the propoied course of artion in 
the long' run help or hinder, ibuild or destroy, .the optimum ItffP ; 
That cannot be done peActly since it means looking into the futur^ 
but certainly it dan be done far better if ^e set our minds to it. 
3/ "dU third and crucial criterion is the test of personal mbsHtutidn. 

■ Are you willing to put yourself in the ^lace of anyone^ls^ affected by 

; the proposed course of action? Theorists and leaderj^ay ve^ welL 
^ think ^ey know what is^oo^ ipr everybody else so l6ng as it is the 
otfier fellow who is affected, but n^urse of action is gqpd unless it 
is good for both leader an^i led. In a dictator state, would the driver 
be willing to take to-^^ of the drivenP When tiie court in a recept 
. trial gave the persons con^ted ol communism the choic^etween 
* ing to jail or going to Russia, ther chose going to jail mis tiiird test 
' is obviously the golden rule, with the phrases reversed. (Is the leader 
twilling to have done to him what he proposes to do*to>tiie led. Human 
experience has found no better test. , 

I make no apology for stressing thege points. The events of our times 
have forced them upon usi It is the espeod^ province of psychologists,, 
of all st^ents and rosearphers in tiie human behavior sciences, eeono^ 
mists, sOQ^ogists, political scientists, historians, Journalists, but es- 
peciaby pathologists, to assume responsibility fbr closftg die gap be- 
tween right and wrong use of scientific knowledge. In his recent annual , 
report. Dr. James R. Killian, president of Massachij3ftts Institute of/ 
Technology, urges a course haH engineering and: half humanities anc^ 
social sciences and stresses the clear need for a larger place in the cur^ 
riculum for psychology. But promoting researched providing instruc- 
tion in these fields alone will not achieve the rpsult.^ The key lies in die 
spiritual survival values, in imderstanding, appreciating and cultiyatiijig . 
them. - 

And tiiis brings me to the third point in my program of action: Tmn 
young people in mMng mlue iudgmenis as pirt of our educational 
ir/rfem~not necessarily as separ^ate courses, ajthough one course might 
,>vell b#lstablfehed, but as parts and aspects in all, courses dealing with 
/human problem wnd what courses do not? It may well be not only 

■ a part of coubes in the human bqhavior areas— psychology, economics, 
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BOi^ology^ htitoi^ and mass cQi^mbnication, but aW^ 
i^^jp ^^Wuod and Biologl^l sdencei. Training in mak^ 
Jgntofe may foUow procadurA sudi as I have just outlinjey; Ri^t 
la judgmmti a^d attitudes ^^^^ developed unless constibus^ 

ii iMadeg and ^di ttie materials in air areas of study and 

Ow )iay-tpHday behavior is k mixttir# of ( A ) ?8i^nta^^ pwiposaly 
. ; dirtotpd ^Otigjitft iiM and actions^ (B) inyoluiita^ extertiall}^ 
; and f bjidyy jtoeqted atfitudea^ and actions^ and (C) eonii-*' 

tic^e^ hrt^ttiat^ thbu^ attitudes, and actiom When *'A^ and;*3'' 
^^r^e^^ed <rft§p CT^ ^wn tothf "C" levd andrauter- - 

; maUbally cbnditibn olir lives* /*A^ Is tiie avenue med in a free society* 
fte 44iip used ;n a^ctatqrial society. It is; at the ^^A*" level fliat 
^ we as f^ee mdiyiduiJs have- maximum tiontrol of o'ur owfl Uves, 

If^iha present rata of school etirollment continues for ariodier gen-^ 
; ; era^cfc^ ei;^t out ol ten adults wiU have gone (o high school ^hd three 
; put of ten wiH have gone ta coUege* Widi this upsurge in sdiool attend- 
moe^ It ^ all i^e inore urgent that education in Idng raftge spWttial 
survivU vajue$ beajme^a part of fta cultural climate of our time. It is ' 
■ shocking to disdover the utter illiteracy and immatunty in die thinking 
^ concamihg ^e durable ^human values on the part of sO'Called edu- 
. catedrlntdllgent people; ^ . . ; 

' Heife Is brie example, ivhy shoul^nyone get mdre dian $7500 in- 
come a year? Analyze tfi^ ti^ Apply ^e criteria I listed a mo- 
ment ago* r ^ 

1. "Wiiat wouM be the effect on society as a ^ole? Oh IndiWdWtf^' 
; effort and incentive? ' ^ - * ' ^ ^ 

2. Woidd tfiese effects help or hinder life at its best^ith optmium 
freedom of choice for everybody^ not merely for the individuals con-. 

^^aemed? ■ /. , " ■ 

3. Would the enforcer of ttie idea be willing to exchange places 
Myone else affpcted by the planP - ^ 

To tiiink it through thoroughly and objectively would be good Gain- 
ing in value judpnents -not dtdy for teenagers but for mmy>adults,\ 

Take another idea— not prevalent today but held stubfcbmly a hun- 
dred or more years ago. Owners and managers of textile miUs made it 
so difficult for adults and children to hold Abir Jobs In die miUs that it 
was next to mppssiUe to get time off f dr children to gp to school, 
^ Ag^ apply, ^e same 4ree tests. Tlie stubborn obstinacy on the 
. part mfll owners led to tte fpimation of labor unipns — and ftat 
was a good fting. It was possible ^CTeby to tiirow off this op^essive 
yoke. In turn, however^ power bf tiie um'on'feaders led to extremes on 
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\^ iother side, Ai' always , 
genariatei counteracting 
and abuse on the ottier 
As I look back ttpon 

■ and higlMy yespeetfe 
:aWay f roiti value jud 



somettiing on a level 

Bctoc^ has its hancil 
true noy?nly in the physic 
Look at ffija subtle minded 
the fruiff^ion bf the frefe spi 
i tfie last 2^^rs. Wars today are 
won ih men's tfihids Before tl^e| 




lns/Exctoi^^ power at oria extreme 
m tvm lead to ex^sive power 

I an^l^hing days in sevW large 
.recal^ow prdfessor^ould shy 
iraiimtff^irifeal-^nsiwrati^ 



life g^d death. This is litWaily 
^ th^N^chological area as wfell. 
:hd the brutal brainwashing and - 
„ that we have seen going on in 
\vbn on the battteBeld. They are 
.^^fting in the fle]^ e^n starts -I 7 
pose, therefor^ to you p^ch^ts especially intkeited^^ta 
tiorial testing probtenis^wd tbirffc [ 

L.That >Su urge Se neecf ^feining in yahie judgments as part 
of our edutatiorsi prjpcesse^ ajiS , . j ^^^^a o^h 

2. That Mou devi^sts, for ^easuHng value judgments^ a^^ 
tude^ and ^ngas produced ty sftchtraini ; 

The far^reaching effeet^pW* any ilUconsideM not to. say irresponsi^ 
ble decisions of men wh^ govern so greatly ouhvei^ the passive 
goodwill of flie mmions wlio suffer therefrom t^t the only cure can ^ 
^ ttie goodwill of to rnilte 
ing and Cultivating; valHe jSdgments and attitudes. A dictator can do 

wktli'stfokeoEVpenw^^^ 
Mes. todo anyt^ Atef Bchweitzer recently said, 

"Scientists rnust speak up." ' \j ' ■ 

Aiid now to summarize briefly: • - ,i i j * 

1 Recognize that knowledge of itself does not automatically lead o 
wise ■constructive use and that therefore we must set about speciflcany 
to seek a6d promote wise use, , .j a 

2 ■ DAemine what is wise use .by evaluating proposed ideas and 
CQurses' of action against three oriteria such as I have ourtined^ 

" a. The test of oduse and effect. ' ■ > 

b The test of spirituaUiirv^al value. 

c The tf St of personal substjftition between leader and led. 

3 Make the training faf y4ng ^people in value judgments a con= 
sciously recognized part of ^ut school and college telohing. ' 

It is evider\t what will decide our destiny,, our ways of livmg, our 
happiness, our well-being or our milbry, wjll not l^e miracles of 
science by themselves but the stamina of our .wisd&i m usmg this 
k^o^yle■dge. "Wisdom and knowledge shall be^tbe stability of thy times, 
and strengtii of salvation." , \ 
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/ SdmQtuiias a topio becomei so involved with situations emotipnaliy 

, charged diat dispiisipnata corisideratjon of it s#eins aknost imposiible. 

Oooasioiially^n unfoiiunate label leads diinldng asfray. Educational 
r ac^aration is a subject which hm iuffered^peatly itom 4ele handi- 
oaps«and^inore. It was dosely associated with flie war and two clumsy 
expedients ^en:; college ehfrance without completion of secondary 
sc^pol, which aroused bitter antagonism in public school people; and. 
" leng^ened college y^j* pwhich burdened arid antagonized college 
faculties. ^'Acceleration^ implies ^uny widi probable superflciaUty, 
And every experienced edticsiter has knowil bright youngsters double- I 
promoted^tf an older group ;who there felt miserabfy out of place- 
perhaps had been such a case, ■ ^ 

In contoast, tills paper will beg your open-minded epnsideratiort of 

. possib^ty f^a^ for abler studenb'^o progrQss^t^liigh schoql at ^ ^ ^ 
faster=tiiaii-average pace is normal /or .^ftern, not hutting; tiilt ttiere 

. are ways of *facffitating their {irogreis which help rather than hinder 

good social adjustmeiiti and/tiiat such steps can lessen flie load and ^ 

J facilitate the work of our overcrowded schools. Art additional gain ^ * ^ 
should be very timely; Russian universities and tedhnlcal schools 
appear now to-be graduating^ about three times as many engineers 

^ and twice as ttiany scientists a year as American institutions** Multiple 
evidence indicate tiiat facilitatirig die progress of able studen treads 
more of tiiem to, complete collegiate , and professional trainingp^o 
probably the oecasional notable genius (die Edisbns or EinstcinsT* 
wiU thu^ be more likely to reach full, fruition* * ^ 

I'wo Mftjor Neglected Facts of HumaoBev^lopmeirf 
Of bafic importance are two conclusions ffon^ del^spmental ) 
studies • regarding die gifted — in childhood and youth, arid in adult 
career. First, gifted children tend to develop more rapidly than die 
' average youngster, not only Intellectually artd educationaHy b^t al|rf 



4r - * M^^off, A., Unttad States ^ortagei SdintistSj V. S. ^ews^and World 
He^Tft, Jantia^ 15, 1954, 46-9. . , * 



< 



108 " / 19S4 IN,VITATJ0NAL CONFERENCE 

phys'&Uy and in persJiiaUty. The bright six yea| old is likely to be^not 
only in inteUigenceiiit also reading and phywque and social assur- 
an4 more like a second tha^ tot grade child The bnihtsixteep 
year old is probably not only in aWUty.and in g|neral-knowledge «p 
with the genertl run of «ighteen-year='old high school senio^ or ewn 
^nfarteen-y^d-coUege^fa^msDi, he 
earlier tiian average ^is In physique; interests, and social adjusbnent 
more Mature tiian tirf^erage for his age. To haVe hiB^^m Seventh 
or twelfth- radier than the tendi grade, or to start bright six 3«ar 
, old In Ae second grade, Is not to hu^ ^Un but rather to hpve hmi 
progress according to- his real growth rate. Terman remarks tiiat, The^ 
exceptionally bright student who Is kept with hfe age group finds ; 
little tb diallenge his IriteUlgence rfnd tpo oftfen develops habits of lazi- 
ness Aat l^er wreck his college dareei. I could gl^fe you some chofce 
examples ^kis i/i my gifted grotip."* There Is also evidence that hold= 
Ing a bright youngster back with his a|e group is lesrf favorable to good 
social adjustment than carefully advancing him Into a i^up more 
like him in ability and maturity of perspnality. ' . - : ■ 

' A second majo* ques*n of .educational policy r^gltrding the able 
student Is ufually pot fleed squarely- if he is to b^ most pr^durtive 
In carett- and inost adequate as a citizen.' at about what age should he - 
be throughtWith fulWme sdiool and reany begin hi adult hf ePVital 
statistics s^w the late tee^ and early , t^^enties to be the hea^|st 
yeatt- rtlysical tests and rthletlc records show them to be physically 
the fflMt A^igorous. And iehmail's , very extensive findings regwdlng 
agei "of m^st brilliant scienti% discoveries, most important Inveiiyon?^ . 
best Siting most remarkable paintings, all indicate that the most o% 
* standing creative work Is done' early Irf.the adult years-often In ^e 
twenties, solnetlmei . even in the teens. ^ The total, of such evident 
is impressive. Nevertheless, American educational programi for our 
most able young people' are bf ing more and more lengthened^ they 
^ust have not only oollege^ul also graduate or professiofta'^aming 
*• perhaps- an "Intern" ex^ienbftiposslBly even some poif-doctoral 
work Before the seconS^rld war, the median age oFftcelvlng the. 
((■ Ph.D. degree in this country wa^qO- now. It tends to be a IJltle later. 
^ By the age of 25\ Efiisoli and Elniein were doing impa^nt creative, 
■ work If Aey wbrk of that a|e in Ms' country now, the/ wguld Instead 
probably be worrying about Iheir language requirenientsl However, 

' • ' 'Tinnan L. M. The diicov4ry .■'and oncourogemont of excepUonal talent , 
' • ' Am^P^ihdig^mB^, 9, 221-30- also The gifted chM grows up. |tanford 
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• E^en ml^t'have h^sk q^rad ju^ diipcidties; he would probably Qot 
^vtf bttn ^baitted to m j^&tjmsL ^ns^^msityi not*ha\^g>€y0n baan to 

■ ; " lUe' argiuaiot thus is that^/^ ^u^mis should f^figresi more rap- 
V wy ihM :tM looJmt^ r^e mrough ^ihdol and cMege because thfiy 



<n#a tfttfr prod^iim mreers earlm'^i^m oaaii^ vHth{the tochitep. 
_ * ^ But what about possibtf soeial maladjustaieDt^ gaps in training, or 
' damage to hedAf ^^ ^ * . ^ \^ ' 

• ^ ^ ^' ^, 1^ ; ^ . . ""--^^ ,^ 

. Means of "^Ac^r^^ation^ ^ #. ' 

By a histortol pervi^ityp the woi^t means for rapid progress— 
grada^sldpping in sc&ool arid the lerigtli^S^ s^ool year in coUfiia— 
_^ have been so muph more used than ^tter J^ethodl Aa^ to many 
* ppopl% ^ese wo^t jivays are synonyrnou^wi^ '*afeceleratipn," The Jyay , ^ 
/ who is sWpped fr^o^our^ grade to sfaft 
* Y is less mky) mayfsuffer at le£^t briefly f^m igno^^e of some aritfi- * 
Q metic {^ro^s taken up iii*4e flfA grade. If he is a conspicuous only/ 
, qji© &US to be advanced intb a close»Iaiit Jittle grade social unit, 
he may imtially meet ^tiUty— apd parents may talk. But eved thii^ - 
y clumsy mediod — OTfde-sIdppingl— need not cause much troubW, 

Bright diildren are^ usually aheap of tiieir age in tiheir readfeg^and ■ , 
othfr subjects; a little help from a teacher or parent usually taKes^care 
of*any omissibtts. If grade-skipping of bright pujpils is mad^ f airly^^ > ' 
common and, if (as is common ^nowadays^ changes in tfie membetshipf' ' 
of a ffi>ad e p^oup are frfquent for other ^easoAs, social difficulties are 
usually minimaL Trouble is likely^ only if 
' really of superior ability but is^ushect afej^ad because of parej^tal i ^ 
" , pressure, or school \^ork neurotically superior in compensation for a 

social maladjustQaent alre&dy existing. Grade^skipping is the e^dpt^ . ^ / 
method of advangiag a pupils ^ especially in a small school; ^n<Cff ^ 
"skippers" are selected* carefully on i^e basis of adequate measiie-* f 
^ ment (and a UttJe help i^given in adjustitif for the skipped work and . 
to the new gr^p), outcomes are uwally good— as will be seyn shordy* ^ ^ 

• However^ there need be no skipping oMer possibly needed school 
contend or moving ah occasional diild^into a strAnge group, Chil^bn * 
may be admitted to die first grade (or transfeired from kindergarfeh) ^ 
on the ba^ of mental age and read^ readiness rather dian chifepd- , v 
logi^ age; and some h^-dozen investigations have all ihown briglil 

five year 6lds, so admitted, diereafter excellent in school work and in V ^ 
relations witti oAer ^lldren/ Or bright six year olds with sbme ^itial ^ 
skill in reailing may be started in the second grade. Or, a ?primanr 
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poor may Wovs?" together all duldrfln usuaUy in the first toee grades, 
and move each on into tiie fourth grade when he is ready, ^rge junior 
hlrfi schools may have rapid progress seoUons. made up of bright 
youngsters with excellent school records to date and good sooml ad- 
'^tment. which, ^ o aia usual three years' work in two. And several 
iarehil 'investigafe».h^e shown thlsenfHpld=progress-pupilS-doing-~ 
as well lA senior higTs Jool. academically end in relations widi other 
students, as pupUs of 'the same general ability ancJitetBl record at the 
beginning of junior high but who spent the usual tiiree years there. 
"Similar rapid progreis sections (three years in two) in senior high 
have shown similar success in odlege. ',,1.1 
A few college students may be rBaUy fatigued by a lengthened school 
year though a total bf six we'elcs or more-of vacation usually still re- 
mains; both fatigue and protest seem greater in the facul^. A few - 
students who need the earriings or th&^xperlence of summer employ- 
ment may miss these benefits. But the questlonableness of a double 
assumpflon In the i^Aod must.be stressed u that bri|ht students can 
learrt only in college courses}\and that they must, spend as much time 
In these courses as averag^; students. A substantial number of studies 
have shown that students who obtained o-edit for a. course by passmg 
a comprehensive exkmlnatioS in it instead of .going to class (prepara. 
tion having been obtained from superior or extrn work in secondary 
school from indepehdent stiidy, travel, or^therwise), do exceUently 
in later courses In that subject and also In total college record. Stream-^ 
lined sections of college courses/with methods adapted to the superior 
student and with reduced hours In class to facilitate acceleration, have 
.been reported as successful, also honors programs combining or re- 
placLng sevbral courses, and guided independent study. In short, the 
proa-ess of superior youngsters man be faomtatedin a variety of ways; 
most 'of them are better thart grade^sklppihior lengthened school ysar. 
It need hardly/be mentioned that wise use of each requires initial test- . 
ing to assure su'perlorlty, and continuing measurement thereafter to 
^gulde progress and determine, outcome. ^ ^ , u i. 

Whatever the methods for rapid progress, modifications should be 
attempted^ of the psesent powerf{d social pressures for the lockstep. 
Thus from the day of enlrance, the coUego student is designated a 
m^iaek- of the class which is to graduate four years later; the entrant 
in '54 is a member of the class of '58 (English universities are wiser— 
thera an entrant in '54 is a member of the class of '54). Each of the 
four years Is given a distinctive name and status; special oppprtunities 
(as for presidencies tof student organlzfltions ) me open'only to students 
in their fouMlj, year. A student who finishes college in three years soon 
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Becomes g social anomaly rwhb belongs ixowbefe; Iijstead the feeling 
should be fostered of belonging to the schopl not the class, associa-^ 
tions and friendships should be freely formed within the- total student 
body without class distinction/ And ' status should be determined by 
academic progress and other aecon ^ pIishmentSy not time served, ^at 
would be the demoo'atie way; and it would benefit not only the gifted 
but also tfiose who for some reason take Ipnger than the usualHinie to 
finish an academic program.' - " ^ » ^ 

Outgomes of **4cceleration"'— and '*the Cold War" 

% As already indicated in passing, the evidence is that most fable itu- 
dents do riot suffeV from acceleration/ But much evidence goes further. 
They actually seem to gain! When Jerman compared those in his 
gifted group who graduated from hi^ school young (under 15%) 
with those/^graduating near the usual age (ovdr WW) found that 
16% more of the you^pr group gr^uated from collegd, and 19% 
^ more took one or more years of graduate work. The young group mar- 
ried dtver a year youhger, and had fewer divorces. And twice as many 
of the younger group were highly successful vocationally. Results to 
date of the Fund for the Advancement of EdTtcationj and of the Ohia 
State University investigations^ are similar; these last^tudies show bodi 
tMose entering young and those taking a four year program in three 
years doing bettor in college, and having more successful careers 
a^er^ than matched ^ses of the same ability but entering at the usual 
tim& or taking the usual four years for a degree^* And in most of the 
investigations to" date the accelerates were not carefully » selected as 
capable of rapid progress^ or given guidance in acceleration: alsQ, 
-they moved forward mostly by thes# least satisfactqry methoHs— grade- 
skipping or lengthened school year. There is every reason to suppose 
that better selection^ guidance^ and methods ^uld bring yet better 

^ outcomes. For instance, 25% more of a group of bcight freshmen se- 
lected as capable of acceleration and given guidance in so doingj fi- 
nally obtained a degree tlia^ra controlled group not- accelerating, A 
more effective educatiop^^n^^Zeas tinie—liow can ihisCbe explained?. 
Presumably becafee/Sore ra^d progress is nonnal for the gifted, and 
the beginning of adult career (and marriage and responsible citiien- 
ship) without long delays is ifloibgically sound. There are of course 
certain obvious ^vantages, as that saved tin\e and funds' leave more 
of both for advanced training. There are also motivational gains of 

* Thp Fund fpf'the Advancament of Education. Bridging the gap between school * 
and college, ^J953. Pressayj S. L. Edueatjonal acc^rationi Appraisals and basic 
problems. OfuV State Univer., 1949. ^ - 
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possible pertaive importance. It is to ttie American ttadition, and stimu- 
lating to the bright and ambitlou's youngster, that there be opf'ortu-^ 
lufy to get ftheadipthe loclstep negates jilT-that. You piayMve heard the 
jibq that a certain state penitentiary had a malOT advAtage over Ae 
^ei^tartog^i^Versifyi-you^ulJigeto^^ 

well. Wise meam fbr''^caeraHorf a>pi%«i^flte o0ortu^ties for the 
Me tafprge ahead^whidy Jncoufoge a tmeratclimate'Bf enterprise 
and Uvily effort.. ' - ! V V ^ ' 

In R<^uni4 and Final AppUcation 
Ntimerous studies of human ^' development tiius show tfat able 

' youngsters should progressJn schd*faster than tiie loekstep rate pf^ 
grada^a year, not only becausp they develop faster but tdpreventtod 
long draWhout education fpm delaying productive . 
ods of acceleration expedite progress without hampering social ad- 

^ justadnft WiA. acceleration, more able stildenti* complete ^advanced 
training; tiiBy^et Into productive careers earlier. And Aep careers 
tend' to he idore fruitful Jn addition, congested schools are relieved to 
some degree^ It ^ems k not unreasonable estimate that each year 
tiiere remaiif jri tile Secondary schofls perhaps 300,000 bright young- 
sters' who ou^t well have been graduated. More trained mm^ soomf, 
at hss cb^, to couttter the Russiai^technole^eal direatl Clearly such 
^ program calli f or ftequent iippraisals from tiie primary years to grad- 

a uate school; Wtouing'guidance, an^ informed educational statesman^ 
ship. Surely, then, the topic is most appropriate to this conference. . 
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College Admigsion with Advanced 
Standing 

J ^ ^ - 

WILLIAM H. CORNOG 



L I A M y 




The Schpol andCoJlege Sttid^ of Adrnission with Advanced Standing 
^originated in' discussions of the faculty of Kenyon College regarding 
•the possibdity of raviiing some of the rules govemiflg requirairients 
f^toj the' ^acheloj's degree, in order to enable very able students to sav^ 
time and yet n^tjose tiie essential ^values of Cliberal arts edbcatidn, 
^ President Chalmers of Keny^ described the plan to friends and 
j associates, in ichqols and colleger ajid in 19S1 a gicoup/^f twelve 
institutions formed a Committee on Admission with Advanced Stand- 
ing. The committee consisted of the administrative heads and repre- 
srttatives of the following institutions: Bowdoinj Brown, 'Carleton, 
Haverfordj Kenyon, Massachusetts Institute of Technology, Middle- 
bury, Oberlin, Swarthmore, Wabash, Wesleyan, and Williams, At a 
meeting^in the spring of 1952, the college presidents and deans agreed 
to invite 'ffito-4hejQentral Committee twelve headmasters, principals, 
and superintendents and in May, 1952, the full committee of twenty- 
four'met tp organize what was tliencefortii known as tfie School and 
College Study of Admission with Advanced Standing, 

The Central Committee agreed tliat it would limit the study to 
eleven subject fields on the college freshihan level, and in the summer 
of 1952; we proceeded to organize eleven sub-cominitt™, of four 
colleges and threp ^econdaty school teachers each, in tiie following 
subjects: EngUsh composition, literature, Latin, French, German, 
Spanish, history, mathematics, biology, chemistry, and physics. The 
subject matter committees were given the task of defining in their re- 
, spective subjects the standard of achievement of intensive courses in 
secondeuy schools which could be offered to the ablest high schpol 
students and for which the twelve colleges could give partial or full 
fli'st-year credit toward their bachelors degrees. In addition the Cen- 
tal Committee established from its own membership a Committee on 
Individual Development to examine and define the qualities of the 
able student himself and to advise the other sub-committees concern- 
ing elements of development not commonly included in tfie mare ac- 
quisition of Imowledge, 
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iMure dose contact with tiie coUege faculties at ,eve^ poml m 
the study, tiie coUege representatives on the CenJ^l Cjiimuttee 
named correspondents in their Institutions in each subject field. IM 
sub-committees cOntinuaUy aought the aiwe of the correspondent 
and kept them infonned of the progress of committee work. Commit- 
tee assignments alone involved the participation of more than a 
hundred and thirty school and college teachers and a^mmistrators. 
' As is the case with all experiments, die School and College Study 
began witti a series of hypotheses. We rAade some conjectures about 
American.education and constructed an apparatus to test thfem and to 
search for solutions to die problems derived from our postulates We 
assumed 1) the oontiniiity of education in school and college, 2) the 
virtues of the liberal arts continuum. 3) the mutuaUty of mterest and 
understanding of school and college teachers of the same subject mat- 
ter 4) the immediacy of the need to revise the timetable^ and better 
to utilize Ae time of the. Sblest students, 5 ) the possibUity of describing 
desirable revisions and means of better utilization in terms of specifac 
subject matter definitions in eleven college freshmen course areas. 

Some interesting and challenging propositions may be derived from 
tiiese basic assumptions. We have been committed to the testing ot 
such concepts as the following: • . ^ ■ ■ 

1 That able students cafi anjl should be given more intensive in- 
struction In secondary schools and have the opportunity to qualify for 
admission to college at a level hlghee than freshman entrance m spe- 
cific subjects In which evidence can be shown of strong preparation, 
the equivalent of flrst-year college vi'ork. 

2 That committees of schpol and college teachers can define re- 
nulrement for admission with advanced standing acceptable to our 
bvelve participating Institutions' and broadly applicable in institutions 
of similar standards dnd aims throughDut the country. 

3 That acceleration of able students out of high school after two 
years or three Is less desirable than enrichment of the high school cur= 
rlculum and admission to college with advanced standing at the nor- 
mal college entering age of seventeen or eighteen after high school 

graduation. , . j a 

4 That the advancement of American education demands the 
strenediening of our secondary schools, particularly in those divisions 
in which the ablest students are enrolled, and that the colleges have 
an obligation fo encourage the secondary schools who strive to es- 
tabllsh and maintain high standards of academic achievement. 

5 That sound learning Is respectable and that academic subjects, 
the content of a liberal arts education, constihite worthy intellectual 
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and spiritual nourishment for young minds, if tikem disciplinei are 
liberally and wisely taught, 

6* ^la^ finally, diis study reaches beyond parpchial considerations 
of departpaental regulations and self-intarest and even of collag&jL^ 
p^ee requirCTienb that it may offer a chalienge to America edu^raon 
truly commensurate with the dynamism of our culturej the wealtib of 
our resourceSs and tiia still unawakened powers of our highly endowed 
youth. 

Our comrnitt^s recognized from &e start tiie many difficulties and 
pitfalls involved in dmwing bluaprints and specifications by which our 
principles and propositions covld be put to die test of action. Tliey pro- 
duced definitions of standards which present a broad range of topi^, 
in order to give tile secondaty school teachers freedom for imaginative 
teaching and adventurous plannmg. We have avoided standardizing 
subjects within naiTQw limits and havf encouraged variety in second- 
ajy propams for tibe gifted* We have iried to plan courses whl^ are 
liberal alsjp in terms of the academic achievement expected. 'Hie cimiu- 
lative exteacting of the best practices and highest standards of our 
twelve colleges could easily result in an unrealistiCj "premium" type of 
advanced credit* Perhaps tills has happened in some of our subjects, 
perhaps we have priced t^fff^ut of the market, despite the rich en- 
dowments of our gifted ^fcdenfe, NeverthelesSj we shall not allow defl- 
nitions and laminations to be the sole criteria for the granting of col- 
lege crejdit for school courses. Much weight will also be given to other 
evidence of unusual ability and valued personal quaUties, such evi- 
dence as wlLbe found in die recommendations of principals, gmdance 
offlcers, and teachers, It is, finally, impjortant above all, in our opinionr 
to encourage and challenge die secondary schools to exercise freedom 
and a bold use of imagination in, their teaching and planning^ and to 
avoid as tiie plague "cram*' couwes fpr the bright student 

In June, 1953, tiie subject matter Committees and the Committee on 
Individual Development submitted t^eir final reports, which were pub- 
lished and circulated to the twelve^pollege faculties for tiieir vote. In 
the autumn of 1953, tiie faculties held discussions of the recommenda- 
tions of tiie study and, witii no college dissenting, voted approval of 
die experimental plan to consider for admission witii advanced ^edit 
able students who had received insttuction at tiie level defined by the 
committees and had met the standard set by the advanced ^edit exam- 
inafions to be administerei in the spring of 1954* 

In the fril of 1953 also, schools associated witii tiie study indicted 
tiieir readiness and intention to present candidates for examination and 
advan^d credit in accordance witii Ae conditions* of tiie ex^riment 
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outlined In the Central Committee's published Reports of Cfmm*^. 
19SM9S3. In 1983=1954 about 550 students to eighteen schools were, 
enroUed in courses preparatory to application fo'-f ^f^^^J^ 
May 1954 Aese students were examLned m one, two, and m a tew m 
f^nc^Sae and even four subjeets. TT.e total number of — a^" 
papers was about 1000. In the first year of the expenment, the number 
of ^i^iations which we have been able to adminMer has neees^# 
been totted. TUs year we hav^ expanded the number of exper^ental 

^^^■n:!an^^^iir?4thowm.nyg«^^^ 
secondary schooL over tfie county wiU eventually be offered the op- 
po^ni^^d chaUenge ot advanced courses. We If ow on^y that ^ere 
„e probably thousands of very able students who should be given 
"orltenL instruction in our schools. We know t^at r^^. 
"ndary school teachers and principals are eager to do «'or« 
are ^Ing for their gifted students. We also know that education of the 
Sad as they should be educated costs money. We^shall need here 
and diei soLe allotments tef extra teacher time in those high schools 
^luSke mtensiveTreparaUon. Many of "^^l^^^^y «c^ 
exta teacher tiine for remedial work. Certainly education of the handi- 
capidT^^ retarded and the slow learner, costs money a^d money 
wS^;.e are all glad to spend. We of this study believe in this type of 
rSdTture- we further believe that the provision of such special serv- 
S^Te idicapped is not to be regarded as merely die discha^e 
of humanitarian obligations, nor even to be justified to ^^f^^ 
te argtnts of sentiment." The handicapped are -embers ^ 
society & Ime men and should receive these services as a tothright, 
and Lliafe services this society may not take undue credit, as for 
s^e sQlar or added grace. By the same token, we ho d that it « no 
irdemocratic to provide special educational services for^the gifted. 
Thb pSvlsion is also their birthright, for democracy has the response 
S'^^ o afford opportunity for full personal development to aU ^rts 
c Szens and to e^h of them In ways and degrees commensurate with 
Se peris endowment and his aspiration. If this is truly the extent 
of our society's commitment in education, we face the task of discover^ 
1 to boardl of trustees and boards of education how far short we are 
faUing in meeting tirat commitment, and what necessary and moder^ 
Itely expensive sfeps we must take to give our gifted students as full a 
measu^of thiir educational blrihright as we give to their less en- 
dowed, and no more than equally deserving fellows. , , , „ „ 
The advanced examinations which our study admmistered last May 
were given not only to the advanced credit candidates in our eighteen 
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secondary spools but also to college control groups who had had com- 
parable instruction in our selected subjects in the twelve colleges. 
These examinations have been read and scored, and while correlations 
of secondaty school and college performance must be based on some- 
what meager numbers it is a fair general conclusion that the secondary 
' school candidates made vety respectable showings. This is the more 
remarkable since it was necessary in many of our schoolSj in order to 
get tfie experimisnt under way, to make rather hurried revisions and to 
reach some compromtoe solutionSi The schools are confident that with 
a few more years' experience a much more satisfactofy job can be done. 
No infomiation is yet available regarding the numbers of our school 
candidates who have received advanced credit in colleges this fall. It is . 
perhaps significant that many of our schools report that their candi-' 
dates will probably not seek advanced credit, but are content with die 
satisfactions of their enricl^d secondaty school experience. This fact 
recalls the statement made in the report of our Committee on Individ- 
ual Development: ' '* 

"The Committee wholeheartedly supports the values of a liberal 
education and would caution school advisers not to let these oppor- 
tunities for advanced work. become vocational i^i primary intent, Tlie 
gifted student should focus oa the academic challenge and the possi- 
bilities for personal enrichment^ and not be urged to think and aim for 
possible future professional fields too early in his development. We 
recognize the fact that gifted students are apt to feel a sense of 
urgency to get on with professional study, and may therefore look upon 
a plan for advanced credit as merely a time-savings corner-cutting de- 
vice toward that end. We have been aware of a tendency to regard 
both college preparatory courses in school and liberal arts courses in 
college as mere stepping stones of credit to the happy shore of a pro- 
fessional course of study and a career. The saving of time in school and 
collegCj though often important from the point of view of family fi- 
nances and also by reason of the extraordinary pace of development of 
an individual student, is not the essence of this program for advanced 
credit. The essence of what we seek may ratlier be expressed in terms 
of compressing the school and college experience of the vety able in 
such a way as^to make its impact and meaning continuously more 
challenging and rewarding^ and to awaken and constantly to stimulate 
the reasoUj imagination, ethical SJ^nsibilities, and capacities for moral 
commitoient and action in our highly endowed youth. 

"We deplor^e but accept the necessity of measuring scholastie prog- 
ress by credit counts, but we do not accept the mere acquisition of 
crpdit as the aim of our plan. We do, however, hold that credit demon- 
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strably earned 'should be awarded and not be withheld because of 
scruples of academic bookkeeping. Our chief concern is that the al- 
lege ^edit earned in a secondary school course represents an expto- 
enca in personal growth as well as a mature grasp of the subjecritself. 
It should^ finally, be emphasized that our examinations ar^ but parf^ 

^ of the attempt to deflne courses and standards. The cental purpose of 
this study is' to promote and develop courses and teaching in the ' 
schools. To this eftd, we have extended tlie responsibilities of our ex^ 
aminlng committees to visiting ighools and to the^holding of evaluating 
sessions of school ^nd college teachers. One' of the major afflrmations 
oi this study hal been tliat almost incalculable good has adcrued from 
the prolonged and frequent association of our sehool and college 
' teachers. In late June of this year we held tiiree conferences of our 
teachers,? one in chemisfry at Kenyon Ctoege, one in biology at 
Wabash, and one in history at Williams. The enthusiasm expressed 
fey the teaejieri who participated lends support to the final statement of 
our comftiittee on Individual Development i , - 

"We are confident that school and college teachers who undertake 
to share the re^bnsibiUty ancj dehght of educating oiir gifted students 
as .they should be educated will not 'only contribute to the founding of 
strong/and mutuallyTjenefieial school and cpllege ^^^^^^^^^^^ 

ghelp to establish a standard of education in which the natfon 

pride. ^ 11 

In summary, The School ^nd CoUege Study plan will, we hope, 
afford one means of achieving a balance of oppo^nity in AmefiQ|n 
education by making a more ne£|rly equitable provision for a coni- 
.paratively neglected minority in our schools, our ablest students. It 
will also, we hope, help to place a new^imphasis on certain neglected 
or unfashionable necessities in education. Among these are: the neces- 
sity of patient effort in leaming, the necessity of mastery of pne s sub= 
ject in teaching, the necessity of addressing ourselves to the study of 
worthy things, the necessity of producing thoughtful men. It may be 
that we can produce^thoughtful men by a curriculum that does not re- 
quire our students to think, or that conditions them mesely to agile 
adjustaents to the trivialities of daily living. The burden of proof of 
this possibility lies upon those who maintain that the propeF study of 
mankind is how to live by bread alone, in personal security and un- 
assailable ei^epsia. I am unpersuaded that the majority of our students 
are fit only to be taught to earn a living, salute the flag, make orderly 
exits from burning buildings, and come in out of the rain. To base 
education in any large scope on tlie assumption that common men 
cannot be touched by greatness or uplifted by genius is to conspire 

/ f 4 



Nesting problems ^^^^^.^^ 119 

with degradation and embrace despair, What is required to bring men 
into touch y^ith greatness iSj of course, superb teaching. For men must 

''be worthy of the things they teach as well as of the things they learn/ 
The School and College Study svould, finally, express the hope thaf 
education m America can rp-dedicate itself to the making of master 

^ teachers who can make us thoughtful men. ^ 

In Mar^j 1884, in his twenty-first year, a young man who aspired to 
be a teacher wrote iri his ditry, "Out of tlie shabby, squalid, starving 
Jtfe if i t^ome with scholarship about me and in nie and^ on me, these 
are the ^things I s^ioul^ long to do." He meant, to lecttire on Words- 
worth, Emerson: Shakespeare, Dante, Goethe. The young man joined 
the faculty of Central High of Philade^^ia in September, 1884, and 
for t\vjenty-three years was a master teacher. He taught Shakespeare, ' 
and wrote the deflrtitive ten-volume life of Benjamin Franklin^/^ He 
delivered the oration the dedication of^ianklin's .statue in Paris, 
Whi^n news of his ' untimely deatlv reached atratfurd-on- Avon, where" 
he had spent many of his holidays, JV^e local sclio^j/ closed for the day. 
His influence upon a generation of men hiis endured with them, and 
the mark he made upon the school is inerackcable. His name was 
Albert Henry Smyth, and he left a legend and »legacy. I suppose thd* 
moral of the story is this: He was a graduate oniy^of the Central High 
School; he never went to college. He was the son of qn unsuccessftil 

, carpet salesman, but much, was expeoted of him. At the High School, 
he had to study, was given copious opporturn!ty to study, high an§l seri- 
ous things, under teachers who were worthy of the noblest calling, 

. We still have such teachers today. May their tribe flourish and in- 
crease! 
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Special Treating for Abler Stud&ts and 
Its Rela/ipn tp National Manpower 



D A' E L W O L F L E . ' ' ( 



In the year 1820 the pfofessorshifr of astronomy at Trinity College, \ 
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v^ich carried with it the title of Royal Astronomer of Irulaml fell 
vacant Aft^r dua deliberation, the electors chose William Rowans 
Hamilton to this distinguished post, and to research opportunities 
it 'provided. The appointment Avtis a notably successful one^for Hamil- 
ton served widi distinction until his death and came generally ^o be* 
acclaimed as one of the tv^^or tlii^e greatest rtiathematicians^f the V 
English speaking countries'. - ^\ # \ . 

The astdnish^g feature of Hamilton s appoimment was his ageand ^ * 
statusi he was '21 years old and still an undergraduate studeff^ at 
Trinity College. The great boldness of suddenly transform i^&la under^ 
^aduate into a professor marks this as one of the outstandmg^ examples ^ 
of tibe ear5 recognition and reward of high ability. But there have been - 
many other, albeit less dramatic, examples. Newspapers ^Mmagazines 
last summer told the'story of a coed who earned her bapheRN^'s degree^ 
at tiie University of Iflinjpis in a single year. Students who finish dollege 
m less dian four years have bdgn knowii to all of us. Lehman has giled ^: 
evidence on evidtoce of the eariy age at which many great ^ccopipl|sh- 
ments in science, literature, and tlie arts have been ^ade. Professor 
Pressey^s excellent work at the Ohio Stat© University constjtufes ^a 
model of how to bring a larger ^lumber of able students into productive ^ 
careers at a younger than usual age, ^ ' ^ ; 

TTiere is a common element in Professor Pressey's work, in the 
quickly-earned^ degree of the Illinois coed, in Hamilton's youdiful ^ 
attainment of professorial status: in each ca^ a college faculty recog- 
nized and nurtured tke outstanding ability o| a young nian *or woman 
whoae interests, learlier attamn\ents, backgro%md, a'nd financial re- 
sources had brou^t him into a college in which hfs ability could come 
under observation of interested faculty members. It is important 
to identify such students and to provide them, with opportunities for 
tiie fuU development of theiV abilities. But we mult also remember that 
there are other highly capable young men and women who do not go 
to fcpllege and who do not come*under the eye of an interested faculty 
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memb^, Be^us^^^&y lack the xnbney or more fremifently because 
tf^ laye aca4«nic mtereit they go to work instead of WTOlIega^They 
copstitute ft^pgr^on of nation's int^lectual resourges which^e 

, not now\utilizinj| as effecti«|fcM^^e Jyviht * * \^ 

Before taking up thii^ ^^^^Kreater detail, |ef rrie give'a^^s^Yor 

^being con^ra4d^ ^bout ^HIPKll ufiU of the nation's intel- 

lectual j-eWiijces/ TJve ufiffl pJStificatioB c^sists df ^ pompariion be- 
tween anbcipated lupply,and antip^ fo>. scientist^ en^- 
neers^doctprSj 'school teather^ or .professional men and \^(OT^en m other 
fields. These supply-demand co^lparison^^are based upTOWath^r'cyude 
analyses ^ofitite manpow^ market^— drude >^ecause our fe^W^u^ for 

^^^ffirfsating ms^npoy/m donland ate considerably mtor^^^Hniitive thanv^ 
ai-e^ur ^dhniques for estimating a fcqjnmodity dem&d'. Tfiey result in 
§ucfc'pstimatas as ^hat in 1960 tbi nation will be short 4d,00i^^Gt^rs, 
wiM need ^0 and so rfiany more ejlgincers or school teachers thaft. are 
jin'ii^tj^or wijl have an adequate supply of forestto or MWvers.' De- 

* 'Spite tjheif limitationsV such analyses hafe' valuei for a few-years into 

, die futule^^y can be faf aid*in the counseling ^f students ,and fliey 
pfovid©^uide*Jines for the^^^ftiing of eduefttional programs, person- 
nel ^olicieSj and manpower utih'zatign programs. - _ 
, But the^sui^Iy and dei^^ analysis is not the^only justiflcation for 
feeling a serious concern over the problem of how we utilize pur in- 
tellectual resources: * ; ^ . ^ >^ ^ 

"A society which permits a significant portion of its members, to 
work M levels below their capabilities is failing to achieve its full po- 
tentfal stteng^. The ability of a society to progress, the ability to- better 
the goal^ fonwhicK it strives, and the skfll and wisdom wi^ which it 
meets itw^aHenges are likdiy to tHe decisive factors in detennining 
its fate , the wisdom of the socjety is. the wisdom of its members. 
Thus a society can attain full potential only when each.df its mem- 
bers is^enabled to contrifi^te as fully as his individual abilities permit, 
7u(dged by thip sta^^d the Unired States 'has failed fb reach the 
strength \a^ich it'migfit attein, for it wastes the abihties of many of its 
ip^st capable sons and daughters and thereby loses the cdntributions 
mey might have made. Our potOTtial is greattft than our achievement; 
progress in seiende agrf the afts might go on at a faster pace th^n it 
does; the moral s'ffehgth of die country might be greater than it is. 

"niis standard is fundamentally difiFerent from &h supply and de- 
mand' concept . , , Instead of asking How many fobs are there for 
scientists? it asks How much scientific talent is not being utilized? 
Instead of asking How many additional social scientkts and humanists 
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couU be employed it asks Howimich important and useful knowledge 
about hufMntty which we might learn am we failing to leamF^ ' 
Seyiffllfcinds of information are required to answer the question 
how much talmt is being wasted in the United States. We need to ^ 

I know what happens to bright . youngster^ who go to coltege, what 

I hap^ns to equally bright ones who stay out of coUf ge, and how many 
there are.in each group. A full examination ^j^^ questions would . 

, require Agreement upori what we mean by eobd utilizatipn of talpnt 

' and agreemeM upoh the levels and me^ods' of measuring the abihty 
we are^alldng about, I will take up ^ese points .later ih ar published 
account, but let me skip over them herte and give only the highlights , 

^ of a refceiitly complel^ study. , , . . ^ 

Sn operation witfe tiie school and college officials of Illinois, Minne,- 
so2r^ Rochester, N6w York, my colleagues and I in the Commission 
pn Human Resburces and Advaiiced. Training studied a group of^P®^ ^ 
rior high .sch^l paduates of sa^e 20 years ago in order to find outWhat 
they were^ doing and Jiow much tiiey were eating fn 1954. We ana^ 
lyzed tiie differences iaAeir occupations and infeoines in terms of the 
abilily they had shown in high school and in terms'of the amount of 
education tlxey had obtained ^fter graduating from high school The 

' 8,000 dr mom^subjects for whom we had fairly Ml records ca^eirom 
la^ge and smaJlf*"high schools located in urban, suburban, and rUral 

, areas in three regions of the Unjted Stated. \-^- i 

Ability^ whether measured by score on a standardized tpt or by Ank , 
J-high school graduating class, was related to adult occupation. Tho$e 
who In tfie 1930s had bfm^r above average as high school students 
were 'in 1954 . more likeiyNs be engaged in professional occupations 
than were those who were only^slighdy superior to the^average high 
sAool graduate, Moreover, the occupational differences Remained 
when we corrected 'for Hifferertces in amount of post-high school edu- 
cation, Higher4evel positions were associated ^ with higher ability, 
among tiiose who had gone to technical schools, among those who had 

J«terfid college 'bjit had dropped out before graduation, and among 
hose who had received college degrees. ^ 

Similar differences in earnings were found. Those who had flnished 
high school in the top 20 per cent of their classes received higher in- 
comes in 1954 than did those who* finished closer to the high school 
average. And that also was true within each of several educationally 
homogeneous sub-gfoups. 

* WolSe, Dael AnwHcai femurces of speciaUhd tdent New York; Harper and 
Brotei, i954. P, 137. 
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Because fanuly background might well have conteihinated the anal- 
ysis, wb took a look at what ^ad happened to the sons o£ professional 
men^. the sons of laborers, the sons of farmers, an^ the SQP^M 
, other occupational groups, In general, family background^owed a 
small influence on adult earnings and positions. For example, the sons 
of\professional men were receiving .higher incomes than were the sons 
^^abor€rSj whether the cortiparison was made among college grgdu- 
^^""t^lege dropouts, or those who had never attended' college, 
#^The 'dffierences in occupations and incomes which were 'associated 
witfl ^fferences in ability, anj^^ tlxose which we^^associated with differ^ 
encee Wf amily background, were, n>uch smaller than were the occu- 
pational and earning ditferentials associated witK differences jn educa- . 
tion. At any ability level, as ability^ had been manifest in high school, ^ 
and for any background, as ba^^round was indicated by the father s 
occupation, college paid off, Th«| superior high school graduates who\ 
had spent some timeMn college>were better off, occupationally and 
financially, in 1954 than were tliose %^o had not, and those who had 
graduated from college were better oft than were those Vho had en- 
tered college but had not remained Img enough to earn a degree. 
Within the restricted range of ability with which we dealt it meant a 
few hundred dollars a year higher income to be near the top of the 
high school ability distribution than to be near the middle. It meant a 
few hundred dollars p year more to have the background of a profes- 
sional home than to come from a laboring family background, But 
possession of a college degree meant two to three thousand dollars a 
year higher income than was received by, otherwise comparable class- ' 
mates who went directly to work after graduating from high school 
Moreover— and this ppiht I wantjto stress— the income differential was 
greatest for those of greatest ability. If income can be accepted as a 
rough measure of wh^^t a man is worth to society^ then it seems that a 
man of moderately suberior qualifications can earn more and contrib- 
ute more to society with a college education than he can without it; 
a man of very superior qualifications can earn and contribute much 
more, i * , 

These findings lead bacic to the distinction I drew earlier between 
able young men and women who go to college and equally able ones 
who do not, Both groups are part of the nation's intellectual resources. 
If *we Are seriously concemed^ — ^as I think we must be — about hbw the 
nation uses its intellectual resources, we must interest ourselves in both 
of these groups, the ones who go to college and the ones who do not, 
The two grQups are similar in that they include people of higji ability. 
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but the most effeetive techniques of capitalmng tjn Jhair abilities are 
somewhat different, and herein lies a problem of educational and social 
planning, . ' 4^ 

On ^e one hand we Have able young men and women who ^re aca= 
demically motivated. Because of family intdrests, educational success^ 
the aspirations 6f tiieir fellows, or for other reasons, they go to college 
and sometimes to graduafe or professional school* The primary prob- 
lem for diem is. to s§e that they get the best possible education. The 
' primaty danger is that they will be lost in die mass of teii capable ^gh 
school and coUfege classmates, will be offered a l0ss stitaulating educa- 
tional &e dian woidd be best for diem, and will, as a result, waste a 
portion of their time and effort and not develop as far as they might 
with a mbre appropriate edueation. There is a growing tiecognition of 
ttie sw^d needa of Aese students* The progtams described by Dr. 
R'essey and Dr* Comog are specifically^'designed to enable, them to 
make better use of dieir educational yeari than would be the case if 
tfiey followed the route of the average student. 

The other group incjudes those who are bright btit are not academi- 
cally motivated. There are substantial numbers of ^udi persons. At an 
inteUigen^^evel one standard deviati^ above the population average 
about one /man and woman in four ourrendy graduates from college, 
At a level Awo standard deviations kbove the population average, about 
one in two graduStes from college. The number going through college 
icould be inc^easai by offering more scholarships. But financial heJp is 
not the only remedy necessary* Interest is more often lacking than 
moijey. Encouragement, guidance^ and an elementary and secondary 
school experience tiiat will challenge their abilities are at least as 
necessary as are scholarships* To bring about these improvements will, 
obviously, require changes in attitude on the part of many Jseople and 
will require action a^t die local community level The task is one for the 
entire educational system; and for the society tiiat supports and bene- 
fits from diat system, 

In conclusion I would likfe to make one final observation. It is this: 
such evideqce we have indicates diat it is' wordiwhile to give great- 
est attention to tiiose^of highest ability. With limited resources, I would 
concenfrate on^e top one or two or five per cient; with more generous 
resources I would extend the effort over the top five or ten or twenty 
per cent. There are tens of thousands of young men and women who 
arte or coiild be above average college students; tiiere are quite a few 
widi very superior ability; diere are precious few William Hamiltons, 
The widely i^mifying conbributions which the very ablest can make are 
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SO ^niportut Aat eve^' re^onable ^effort should ^^diade to identify 
and educate all of Aem, nation nepds tHe sclentifio progress^ 
t^mnologioil aohievement^ and moriil^stcpngA di|t cad coma from 
me mmd$ c^ite ableit ions and dau^ters, Those of up who can help to 
identify the younpteri^of hi^est potentialp who ^n point*tiie way to^ 
overcoming ^e obitacjei whi^ now keep some of ^s talent from 
coming to full deyelopmentp and who ean help to develop ^e kinds of 
^lement^yp secSridaryp ^and collegiate education viWch ii best adapted 
to tiie traipi^g of hi^ly a^le youA share much of the responsibility for 
determining the future progre^ of society, 

u . '■ : - " • ' 
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pabticipants 

Ralph Berdie, Herbert S* Conrad, Edward E, Cureton, 
Walter N. Durost, Dael Wolfle 

Db, CoNBAjD: AJ& ttiese people whom Dr* Wolfle was speaking aBbut 
_^ttose who graduated from hi^ school twenty years agoP 
Dr. Woui^i Tiey graduated between 1920 and 1938. 
Da, CoNRADi Do you generalize from tiiat generation to the present 
one? 

I^. WoimMi In general principles, yes. In temis of amounts and 
det ails , no. For example, tiie earning differential would change as you 
studied a larga fraction of the popnlntion or studied graduates of a 
different timBPeriod. 
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Appendix * , 

Participants — ^1954 Invitational Contef^nca on Testing Problenii 



ApEiNi, Dorothy VUnlversity of 

North Carolina 
Afflerbaoi> Janet, Professional Ix- 

amination Service - ^ 
AmttANK, J, Stanley, Cornell Univeriity 
Alman, 'John E., Bostqn UnJveriity 
-. Ali^h, Kadiryn »M.> bepaftmeat of Ed 
ucatio^t Schenectady 
Allen, Margaret E.,. Boar/d of Educa= 
• tion, Portland, Maine ' 
Allison, Roger, Educational Testing 
Service 

Alt, Pauline M., Teachers College of 
Connecticut , 

Anberhalte^, Oliver F., St. Louis Uni- 
versity 

Andersoh, Rosa G.,: Psychological Cor- 
poration * » 

Anderson, Roy N., North Carolina 
State CoUegs 

Andebson, T. W., Columbia University 

^NDERsoN, Mrs. T. W., New York City 

Ancell, George W., Jr., Educational 
T'^ting Smdce 

Ancoff, William H., Educational Test- 
ing Service 

Anthony, WilUamj Maryland State De- 
partment ol Education 

Apto., Valentine^ Greenwich, Connec- 
ticut 

ARMsraONc, Fred, Lehigh Uftiversity 
AnONOW, Miriam S., Nevi^ York City 

Board of Education 
AnsENiAH, Seth, Springfield College, 

Massachusetts i 
Ballisty, Al, Personnel Department, 

Philadelphia 
Bannon, Charles J., Crosby High 

Sdiool, Waterbury, Connecticut 
BAnDACE, Herbert D., New York State-^ 

Department of Civil Service 
Barnes, Paul J,, World Baok Company 
Bartelme, Phyllis F., Blythedali, t^evv 

York City 

Bartnice, Robert, Educational Testing 
' ' Service 



Battison, Mrs, C., Remedial Education 

Center, Washftgton, D. C. 
Bayroit; A. G., Personnel ^eiearch' 

'Branch, ApO ^* / ^ 

Bedard, 'Joseph A,, Public Schoobj 

New Britain, Connecticut 
Bement, ■ Dorothy M,,? Northampton 

School for Girls ' ^ 
Benda, Harold, .New Jersey 'State De- 

partinent of Education 
Bennett, George K,^ Psychological - 
' Corporation 

Bennett, Ralph, New York .City 
Benson, Arthur L,, Educational Test* 

ing Service 
Bent, Alma, State Teachers College, 

New Paltz^ 'New York 
Bentz, joHj Sears Roebuck k Company 
BehdiEj ^Iph F,, University of Minne- 
sota 

Berg, Joel, University of Connecticut 

Bergeh, Bernard, Municipal Civil Serv- 
ice Conun^sion, New York City 

Bergesen, B. E., Personnel Press, In- 
corporated 

Blaul, R. EUzabeth, Highland Park 
High School, Illinois 

BlqoMj Benjamin, University pf Chi' 
cago 

BoASi, Veronica, Department of Per- 
sonnel, New York City 

Bogar, Jack, Board of Education, Rich* 
mond, Virginia 

Bogosian, Dorothy, Queens College 

Boldt, R= F., Educational Testing 
Service ' ' 

Bollendacher, Joan, Cincinnati Public 

. Schools 

Bonner, Hubert, Columbia University 

Brace, Susan. E., Archdioc^san Voca* 
tional Service, New York City 

priANDT, Hyman, American Occupa- 
tional Therapy Association 

Bhansford, Thomas L., New York 
State Department of Civil Service 
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BnA¥, dduglas W,, Cplumbia Univer- 

Bmomj^i Donald S., Am^can Tal©^ 
pbooe and Tdep^ph Company 

Brotow, WlUiaaa H,, Biireau of Cm- 
riculum Re^archi New Ypric City 

BRODimcE, J., Lawenca, YMCA, New 
Vorfc Ci^ 

Broc^en, Hub^ E,, Per^nnel Re- 

iearch^fandi, AGO 
Brolyw, GacU R,4 New Ydrk Stata 

Department of Civil Service 
Broou, Richard B.| College of William • 
. * and Mai^ 
' = ^ Brown» Frederick S., Great Neck Pub- 
lic Sdiooli* New York 
BnyAN, Ned, Rutgers University 
: Brvan, Miripm M., Silver 3urdett Com- 
pany 

BucEToi^. LaVtme^ Brooklyn College ^ 
Bmi, William D., Temple Universify 
Burdock* Eugene^ Carnegie Coipora- 
tion of New York 

V Bu^m, James M., Darien Public 

i Schoolsi Connecticut 

! BuR^j Paul J., Bell Telephone Com- 

y pmny ' 

^ BuRKHAM, Paul S., Yale : University 
/ BuROS, Oscar K,, Rutgers University 
Byrnb, Lois A., Temple Univeriity 
Byrne, Richard Hill, University of 
^ Maryland . ^ 

Cadwbll, ^Dorothy, Civil Service Coin= 
> mission, Canada 

Campbell^ Donald W., Board of Edu- 
cation, Newark, New Jersey 
Capfs, Marian P., South Carolina State 
^ CoUeg© 

' Carlson, C. Ray, Air University, Max- 

well AFB ^ 
: CAmjDN, Harold S., Upsala /College 

Carlson, J. Spencer, University of 

Oregon 

Carroll^ John B., Harvard University^ 
CBLLmRS, Peter J., Pathfinder . . . 

The Town Jourml 
Ghauhcey, Henry, Educational Test^ 

ing Sa^ice 
CmmGHnx, Ruth, Antloch CoUega 



Clabe, WflUi W., California. Test Bu- 
reau ^ 
Co^, William E,, Pennsylvania' State 

Univaisity 
CcK3iLiN, John H*,^Temple Univarsit^i 
COFFMAN, William E., Educational 

Testing Servica ^ f -^^.j, 

COHMM, Philip State Teachers Col- 

lege, Montclair, New Jersey 
CtfLEMANi William, University of Ten- 

nesse© 

CoNRAfi, Herbert S., U. S. OIHte of 
Education 

CoNWAY, C* B., Department of Educa- 
tion, Victoria, British Columbia 

CopB]LAND, Hennan A., Atlantic Ke- 
fining Company 

ConNEHLsEN, Johu H., Tufts College 

CORNoo, William H>, Central High 
School, Philadelphia 

Cox, Hcn^ M,, University of Nebraska 
,pRANE, Percy F., University of Maine 

Craven, Ethel C., Polytechnic Institute 
of Brooklyn ^ 

CnAWFORD, Barbara, Educational Test- 
ing Service 

Cra%vfohd, J, R., University of Maine ' 

CnissY, W. J. E,, Personnel Develop- 
menti Inc. ^ 

Cronbach, Lee j., University of Illinois 

CnuTCnFiELD, Richard S., University 
of California 

CuMMi^Gj Mary, Boston Public Schools 

CuRETON, Edward E,, University of 
Tennessee : 

CunETON, Louise Wm Board of Exam- 
iners in Psychology, Tennessee 

CynakioNj Manuel, Brooklyn College 

DAHNidB, Harold, Michigan State Col- 
lege , = 

Dailey, Jf)l|o T., Bureau of Naval Per- 

■ sonnel 

Dalv, Alice T., New York State De- 
partment of Education 
Damrin, Dora E., Educational Testing 

Service 

Daviikiff, M, D., U. S, Civil Service 

Commission 
Davidson, Helen H., City College of 

New York 



TESTING PROBLEMS 



^DaVs, Fitd B,; Hunler CoUega 
Daybom^ Hu^ M., PrnD^Ivaola State 

DmSf Bowan C.t NatioQal Sdence 

{foundatfon ^ 
DiOANj Jam^ W«, Mam^iiset^ In^ 

DmymmTt^ Leiand Aockafsllgr 

Foradation 
DUMOND^ Lcmine K.p Taadim Gol- 

Diei^ G* W», JnttmsMnsl Buiineii 
Madbliiei 

DASiM^ Fa^B., Ed^catfond Test- 

isg Sendee 
Dion, Robeit, CaUfomla Bureau 
DoBBDi^ Jobs E«, Edu^tiond Testing 

Servian - 
DopM-T/ Jeionje Evj P^helogleal 

Corporation 
DRAaosm^ Ansa, Educ^tldnAl Testing 

Sendee ' 

Diuupii L» Univ^d^ of Wiiconiin 
Druz, William^ New Jmmy Qyil Sav- 

ice Departinen^ Iteitpn 
Dt^m, Sam., BiooId>^ College 
DuHNi Frances E,, Brown .Univ^iity 
DuRAK, June C., Callfomla Test Bureau 
DuRosT, Wdter It Test Service., and 

Advisement ^n9t New Kampdiire 
DwYBH, Paul S., Univm'Si^ of Mtehi= 

gan 

Dye^ Hemy S., Educational Testing 
Service 

Ebel, Jlobert L,, State University of 
bwa 

EDRmOTON, T. Dep^^ent of De- 
fense 

EnwARSi, Robert E,, Rensselaer Poly- 

^^nio f nstitii^ - 
ENGELHARTj Mojc D,, Ghicago City 

Junior GoUege 
EpsTESf, B^raxn, Oty College of 

New YoA 

EiTAVAH^ Doniddt Educational Testing 

Ei^Bii Doroth^i Bloom Township 

School Chicago Heists 
Fak, C. T, Edueatiooal Teiting Servfee ^ 



FAANUib^ Henry M.| Human Engineer- 
ing Devices 

Faar^ George I Inteniational Business 
Madiin^ 

FwrnEBQ^ Mounter R,| Befnard 
Barudk School and R^ear^ Institute 
^ of America 

FmLSfTi Jjeonafd S.^ State Univeitity of 
Iowa 

F^raia^ Paul, W^tem Electiic Com- 
pany % 

Ferouion, William World Book 
Company 

Fkmis, F. L., Jr*, Educational Testing 
Service 

Fields, Mrs. Carlylei Croydon HaUj At- 
lantic Highlands, New Jemy 

FiFER, Gordon, Test .Researdi Service, 
Inc* . ' . ^ 

Firai^Y, Warren G., Educational^est- 
ing Service 

Fins, August A., Jr., Columbia Uni- 
versity 

FmmMi Robert B., .Metropolitan Life 

Insurance Company 
Fm'm, Louise, Educational Testing 

Service 

Flanagan, John C., American Institute 

for Research 
FLETciffiRj Frank M., Ohio State Uni' 

vemty 

FoLGEB, John, Southern Regional Edu- 
cation ^ard 
' FoRLANo, George, Board of Education, 
^ New York City ^ ' 

Fqrlano, Mil. George, Board of Edu- 
cation, New York City 

FoRHESTOR, Gerbude, West Side High 
Schooli Newark 

Fox, William H., Indiana Unjve^ty 

FR^BRSSBtij Nofman, Educational , 
Testing Service 

Freeman, Paul M., Educational Test^ 
ing Service ' 

l^EHCH, Benjamin J.^ New York State 
Departoient of CivU Ser^dce 

Fhencii, John W., Educational Testing 
Servi^ 

FRffiDENBERc/ Edgar Z., Brooklyn Col- 
lep 
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FimmdAMf Sidneys Bureiu of Naval 

Fulton, Ailaa W,, Amarito i^d 

eign Taachm' Agen^ ) 
Fulton, B^aa Board of Eduction, 
, Naw Yoik Cit^ . . 

FiMT, Edwafd J*, Univ^ty of Miehl- 

gan / 
GALLAGBffiB, Hanrietta L„ Educational 

Toting iand^ . . , 

QAsmNm, Eric Fp, Syracuse UnlvOTtty 
Gawo§£e, Romatt 8., Yo^araj New YoA 
G^BmcH, J. R., UniVe^ity of Connec- 

ticut 

GiilbmGs*' Frank, Springfidd Trade 

High idiool, Maisadiuaetti 
GuAsmf Robert, Amsrican I^tute for 

Raiaar^ . . 
Glajs, Albert A., Tha Signal SchmV 

Fort Monmouth , 
GoDDAim, W. A.* International Businisi 

Madiines 

GomnJoM, Fred 1., EduAtional Test- 
ing Service 

GoLmTEiNj hm S,i Teachers College, 
Columbia University 

Goodman, Joan, World Book Company 

Goodman, Samuel M., Board of Edu- 
cation, New York City " 

Gordon, Mary AHce, Macys 

Grimm, Elsiaa R** Profesdonal Ex- 
amination Service 

Gheen, Bert F.i Maiiachusetts Insti- 
tute of Technology. 

GpovBS, Kenneth J,, Air. University, 
MajcweU AFB 

CuTHRls, G^rge M., Penn^lvania 
State Univeriity 

GuLLKSENj Harold, Educational Test- 

i ing Service 

^Aimmi, Heinz, StudieMtiftung, Bad 

Godesberg, Germany 
Haoin, EU^batii, Teacheri CoUige, 

Columbia Unlverstty 
Hagman, Elmer R., Board of Educa- 

don, Graew^ch, Connecticut 
K^^^RN, Joseph, New York State De- 

partoient of Civil Sendee 
Harmon, Lindsey B., Na^nal Research 

Oouncfl 



Harter, K,s Amariran Telephone 
and Telegraph Company 

Harvey, Philip R.* University of Con- 
necticut 

Hastings, J* "niomaSi Unive^ity of Illl- 
noil ^ 

Hayward, PrisdUa, Educational Tot- 
ing Service ^ 

Heaton, Kennetii L., Richardson Bel- 
lows, Henry^and Company 

Hstt, Louis Mp, B^klyn College 

HELMICEi John S., Educational Testing 
Service ? 

Helmstadter* G. G, Educational Test- 
ing Service 

HEMPHmL^ John K., Ohio State Uni- 
versity ' , 

Hn.L, Walker H., Midiigan State Col- 
lega 

Hills, John R,, Educational Testing 
Service 

HiTTiNOER, William F., Pennsylvania 
State CoUege 

HoBERMAN, Solomon, Departaient of 

PersonireL New York City 
'Hollev, Clifford S., Personnel Depart- 
ment, Philadelphia 

HoLLis, William H., New York City 

HoLLisTER, John S., Educational Test- 
ing Service 

Holmes, A. F,, ;Royd Canadipn Air 

' Force 

Holmes j Therese, Metropolitan Life 

Insurance Company 
Horowitz, Milton W., Queens CoUege 
HoRTON, Clark W., Dartaiouth 
Huddleston, Edith, Educational Test- 
ing Service 
HuNSieim, Paul, .University of Mldil- 
gan 

Hunter, Genevieve, P., Arc^diocesan 
Vocational Service 

HuBLBURT, Allan S.* Nortii Carofina 
State Department of Public Instruc- 
tion 

HVLLA, Eric^, Hodis^ule IntOTation- 
ale Paedagogische * Forschung, Ger- 
many 

JanowseYj John, Personnel Departmenti 
Philadelphia 
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jAiron; Kadiu, National L^i^e for 
i Nurtisg 

Jefimy^ W* E., lamard College, Co- 

itusbla Univ^r^^ 
JoffifsoN, A« Fmb^toQ, EduoatioQal 

Te^tiog Sendee ' 
JomisoN^ Lewis W*, Perionnel Depart- 
Philaddphia 
^ JpfiniOf, ArAur Mi* UEdvenlty of North 
^ CaroUna' 
^ Kabacs, Gcldle R.| Ci^ College of New 
York 

' KAi^r, Robert* Iduq^tional Testing' 

KABONi BertTMn P.* Educatlonai Test- 
ing Ss^oe 

I^Ts, J. A., Educational Toting Serv- 
■"" ice 

Km^, John W.* MichigM State College 

KiQ^» Joieph G.J American Council on 
Education 

' Kimball^ Elizabeth G., Educational 
Testing Service 
King, Richard G., College Entrance 

Ewnmation Board 
K^Nii, Davidj New York Univmity 
Kra^ATRiCEj Forrest H,, Bethany Col- 
lege 

Koi^EBECK, Robert P., Education^ 
Testing Service 

KosM^L;, Alice F., Washingtonj D. C. 

KosncE, M*, State Teachers Col- 
lege, Boston 

ItoATHWom.^ David Hi, University of 
Illinois 

KtlBiSj Joseph F., Fordham University 
KuDERj G, Frederic, Duke Univei^y 
Kvmz, Alb^ K, University of Florida 
KuSB^BR, Boie E., City College of New 

York . 
KuTcmR, Charlotte, American Public 

Health Assodation 
LANGMum, Q, R., Psy^olo^cal Corpo- 
ration f 
Lan^olm, G, v.. Educational Testing 
Strilee 

LAimN, Wilbur L,j University of Min- 
nesota / 



I^CH, Kent W.i Unlveriity of Michi- 
gan ' * 
L^VY, Sylvia, Queens CoUegt 
Lbboi^, WiUiam K*, Purdue UnivOTity 
Lm, MarUyn C*, Science Research As- 

sodatel « 
I^NNON, ^ger T*, World Bock Com- 
pany 

Lev, Joseph, New York State Depart- 
ment of CivU Servi^ ' 

LEViNEj RidiMdj Educationd. ^Testing 
Service 

Levv, Charlotte, Chunky Chocolates 
LiMBUHG, Charles U. S. Air P*orce 
LmDQUiSTi F., State University of 
Iowa 

Lohman, Maurice A., New York State 

Departm^t of Education 
Loos, Goidgn M*, Educatiotnal Testing 

Service 

LoRCH, Vera, Great Neck, New York 
LoHD, FrederiOj Educational Testing 
Service 

Lord, Shirley, Educational Testing 
Seivlca 

LoHGE, Irvingp Teachers College, Co^ 
lumbia University 

LoRH, Mau_rice, Veterans Administra- 
tion 

Lund, Kenneth W., Public Schools, 
Chicago 

' Lvse, Louis T,, No^alk, Connecticut 

LuTz, Orpha M., SMa Tea^ei^ CoU 
lege, Montclair, New Jersey 

McAhthuh, Charlei, Harvard Univer- 
sity ^ _ . _ 

McCab% Frank J. I Metropolitan Life 
Insurance, f^ew York City 

McCall, W, 6.,, University of South 
Carolina 

McCambhidob, BarbaJa, Educational 
Testing Service 

McCanh, Forbes E., Personnel pepart- 
ment, Philadelphia 

McCoHD, Richard B., Penonnel Depart- 
ment, Philadelphia 

McEwah, Dorothy M., New York State 
Department of Civil Sendee 

McGttL, William J*, Massachusetts ln» 
lytute 0^ Technology 
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MdG^m, Iffiatt M*, Unlverrity of 
Maryland 

Mwir^ AFB 
MoIjAUO^^, Eugfoto, New York State 

D^rtmant of Civil S^nioe 
MaNAMABA, W*^p taternational Bud- 

am Ma^^^ 
MaiJ^aiL; a Brown Uniyertity 
McQmm, John V*, U^verii^ of 

Fbrida 

Macmubo,. Carles J,, U. S. Naval Ex- 

amteing Canter 
Macoi^ Vincgnt 8., Columbia Unlvemty 
MAmm^i He^diel .T., University / of 

T^cas / 
Mi^Bio^i John C*^ World Book tom- 

pafiy ' 

MABSTOKj Helen Edueatbnal Teat- 

kig Service 
Masxa, Be^am B.* New York Univer- 

dty 

MATHEWi, Chester O., Ohio Weskyan 

University 
MathewsoNj Robert H., Diviiion of 

Teach^ Education^ New York City 
MaXsoNp G^rgla^ Idu^tional Testing 

Sendc© 

MraJJYj Donald M,* DivisiSn of 
Teadier Eduoation, New York City 

Melville* S, Donaldp Educational Test- 
ing Serviee. 

MerenoAj Peter Fp* U* S, Naval Exam- 
ining Center 

MmtYj Pobert W*, Harvard University 

Mbtz^ Elliott> Que^s allege 

MiOBASLi WflliMfi B.* Univeriity of 
Souths Cdifornia * ■ t 

MiCBAs^, 8, R.p Educational Testing 
Service 

'Mia»Li^ Genej Metropolitan Life In- 
suranoe CorDpany 

Mn.Ei, Matthew B., Teachers College; 
Columbifi Univ^iity 

Mnx, Cyril Public SchooISp Rich- 
mond 

MflLUKET, Esther, Westover Schrol, 
Middlebun^ L/ 

Mrr^mLL, Blytha World Book Com- 
pany 



Mn^j Harold E,, Dlvidon Teaser 
Eduction, York Qty 

Mou^NKOpr, William G** Educational 
Toting S»vi^ 

MoRGiWf, Antonia BeU, Aptitude Asso- 
ciates In^rporated 

MoRQAH, H/ H.i Piychologieal ^flpor 
ration ^ 

MoRms, John B., University of Minne' 
iota 

Mo^B, Nancy, Educa^onal' Testog 
Service 

MoRmsoN, Alexander W„ Polytedinic 
Institute, Brooklyn 

MoraiioNp J, Cayce, Huerto Rican Study 

MOBTOH, Anton Education^ Test- 
ing Service 

MosELY^ Russell, Wiiconrin Stat© De- 
partment of Public Insttuction 

MuHOBR, A, Mp, Standard OU Company, 
New Jersey 

Murray, John E , Special Devils. Cen- 
ter, ONR 

MVEM, Robert L., Temple University 
Nelson, Kennedi D„ Division of Re- 

icarchj Albany 
NELioHi M. J. I Iowa State Teachers 

College 

NEvmj Margaret, Educational Testing 

Service 

NiLL* Kathryn F., Rinehart & Company 
Noll, Victor H., Michigan State Col- 
lege 

North, Robert D„ Educational Records 
Bureau ^ 

Norton, Dee W., Air Fore© Personnel 
k Training Researdi Cent^, Lack- 
land AFB 

Olsek, Manorial Educational Testing 
Service 

OrleanSs Beatrice S., Bureau of Ships, 
Navy Department 

Orleans, Joseph B„ George Washing- 
ton High S^Dol, New York City 

Pace, C. Robert, Syracuse University 

Palj^r, OrviUe, Educadonal Testing 
Service 

PAs^kLiAN, Skoonp Qu^^ CoHige ^ 
Peabson, Richard, Edu^tional Testtag 
Service 



